Theory of Mind in Large Language Models: What Can AI Actually Understand?

Few questions about AI generate as much heat as whether language models have a "theory of mind". The phrase sounds grand. The research behind it is more modest and, for that reason, more interesting. This article explains what theory of mind is, what the key studies of language models found, why they seem to contradict each other, and how to read the next headline that claims AI "understands" us.

The test that started it

Theory of mind is the ability to attribute mental states, such as beliefs, desires and intentions, to oneself and others, and to use those attributions to predict behavior [1]. The signature test is the false-belief task. In a typical version, one character puts an object in a basket and leaves. Another character moves it to a box. Where will the first character look? A child who answers "the basket" has understood that the character's belief differs from reality [2]. Variants of this paradigm were used to study autistic children, a line of work that made false-belief tasks a standard tool in developmental psychology [3].

It is easy to turn these tests into text. That is what researchers did when they asked language models.

The headline result

In one widely discussed study, Kosinski gave eleven language models 40 bespoke false-belief tasks. Each came with true-belief control scenarios and reversed versions, and a model had to solve all eight scenarios to count a task as solved. Older models solved none. GPT-3 (davinci-003) and ChatGPT-3.5 solved 20 percent. GPT-4 solved 75 percent, which the author noted matched the performance of six-year-old children in past studies [4].

A larger comparison is more careful about what it says. Strachan and colleagues tested GPT and LLaMA2 models, and 1,907 people, on a battery of tests that included false beliefs, indirect requests, irony and faux pas. GPT-4 models performed at or sometimes above human levels on indirect requests, false beliefs and misdirection, but struggled with detecting faux pas [5].

The pushback

Other researchers asked whether these successes reflect a general capacity or familiarity with a format.

Ullman made small changes to a published theory-of-mind task that preserved its logic, and reported that the results flipped. He argued that, in evaluating intuitive psychology in models, the default should be scepticism, with outlying failures counting for more than average success [6]. Shapira and colleagues stress-tested models on six tasks and concluded that while models show some ability, it is far from robust. Adversarial examples suggested reliance on shallow heuristics, and the authors warned against drawing conclusions from anecdotes, limited benchmark tests, or human-designed psychological tests applied unchanged [7]. Earlier, Sap and colleagues had found that GPT-3 lacked this kind of social intelligence out of the box [8].

Why the findings differ

The studies are not necessarily contradictory. They measure different things.

• Format familiarity. Textbook false-belief stories appear in many forms in the text models learn from. A model that has seen the pattern can answer correctly without tracking anyone's beliefs.
• Hidden reasoning demands. Some tests ask directly about a character's belief. Real situations do not announce which inference matters. In a study that asked models to act on inferences about others' minds, models that did well at tracking beliefs in stories did poorly at choosing an action, because they had to identify the relevant inference themselves [9].
• Interaction. Benchmarks that use passive stories miss what happens in real conversation. The FANToM benchmark tests theory of mind in conversations where participants have different information, with several question types that require the same underlying reasoning. Models performed significantly worse than humans, even with chain-of-thought reasoning or fine-tuning [10].
• Breadth. ToMBench, a bilingual benchmark designed from scratch to avoid data leakage, covers eight tasks and 31 abilities. Its authors found that even GPT-4 trailed human performance by more than ten percentage points [11].

How to read the claims

When you see "AI has theory of mind" or "AI has none", ask five questions.

1. What exactly was tested? One task format, or a varied battery?
2. Was it perturbed? Does success survive small changes in wording, scenario or language?
3. Was action required? Or only an answer to a direct question about a belief?
4. Was there a human comparison? And whose humans?
5. What does the author actually claim? Kosinski, for example, explicitly raised an interpretation question, as did his critics. Their disagreement is about what the performance means, not about whether it occurred.

What we think, and what we don't know

Our reading, which is interpretation and not a result: current models can often give correct answers to standard questions about mental states, and that is a real change from earlier systems. The evidence does not yet support the claim that they represent other minds in the way people do, and it shows that performance is sensitive to how a task is posed. For anyone building products that depend on understanding people, the practical lesson is to test the specific behavior you need, in the language and context you need, under variation, and not to rely on a single benchmark number.

We do not know whether better models will close the gaps by scale alone, or whether explicit mechanisms for tracking beliefs are needed. There is some early evidence for the latter in multi-agent settings: in a cooperative text game, agents given explicit belief-state representations performed better and made more accurate inferences about their partners [12].

For the full survey, read Research Report 001. For how to measure these abilities better, see building better benchmarks for emotional and social reasoning.