LLMs can mimic past examples of reasoning from the dataset. So, it can re-use reasoning that it has already been trained on. If the network manages to generalize well enough across its training data, then it can get close to reproducing general reasoning. But it can't yet fully get there, of course.

Do you have evidence LLMs can indeed generalize outside their training data distribution?

https://twitter.com/abacaj/status/1721223737729581437/photo/...

No. I know only that they can generalize within it, and only to a limited degree, but don't have solid evidence of even that.

So what you're saying is there's tenuous-at-best, non-"solid" evidence that LLMs can reason even within their training data.

And yet I'm currently sitting at -1 for stating the blisteringly obvious. Lmao

Yes, that's basically what I'm saying. Just less bluntly. It's slightly more nuanced than "LLMs cannot reason" because lines of reasoning are often in their dataset and can sometimes be used by the model. It's just that the model can't be relied on to know the correct reasoning to use in a given situation.