Many people seem to believe that LLMs "think". Some people believe that the LLMs can "explain" the output they have generated. And some people believe that an LLM can "teach" you how to think.
This is (a) incorrect and (b) dangerous.
(a) LLMs don't think in a way that we would consider logical; they compose text by adding word by word (actually, token by token) based on the frequency these word (token) sequences appear in the learning material. So, the answers they generate reflect all the language in the learning material.
If you ask the LLM a question it has seen 1000 times, it will provide the answer that was given 1000 times. If you ask it a question that is not contained in the learning material, the answer will be unreliable. But it will sound great anyway.
And if you ask the LLM to explain what it did, it creates an explanation that sounds logical. But the answer and the explanation can both be entirely wrong.
(b) You can't learn reasoning from an LLM, because it doesn't reason. It produces text that looks like reasoning. If you practice by following that text, you practice mimicking the appearance of reasoning, not the substance of it.
If you want good answers from an LLM AND if you would like to learn how to reason, one approach is that you provide a step-by-step reasoning flow and then tell the LLM to follow these exact steps.
Why? Because the probability that it has correct answers for simple, small thinking steps in its learning material is much higher than the probability that it has the answer to your complex question in its learning materials.
Why am I writing this? A post was sent to my timeline that suggested exactly the above:
- ask Claude something
- let it explain what it thought
- learn from this.
(I'll add a link to the post in a comment).
What really worries me is how many people applaud the approach suggested by the post - probably because it sounds so logical. Even though it's incorrect.
So, what does happen if entire societies start following bad advice just because it sounds great? - We used to have this (monarchies, religious societies). I for one am not sure whether I want to see another one developing where the "gods" or the "leaders" are a bunch of LLMs...
[Edited to sharpen the "you can't learn reasoning" argument]