Are LLMs Smarter Than Chimpanzees? An Evaluation on Perspective Taking and Knowledge State Estimation

TL;DR

This work probes whether large language models possess theory-of-mind–like abilities by focusing on knowledge-state estimation through two narrative tasks: Implausible Knowledge Detection and Knowledge-sensitive Next-action Prediction. The authors construct two short-narrative benchmarks via a data-pipeline that combines open-source stories, automatic summarization, GPT-5 generation, and human validation to yield challenging IKD and KNP datasets. Across a wide range of models, results show near-random performance on both tasks, with humans clearly outperforming machines, indicating a substantial gap in knowledge-state tracking and intention understanding. The study also finds that model scaling and extended reasoning offer limited gains, while explicit prompts focusing on character knowledge can boost performance, suggesting concrete directions for future research toward genuine understanding of others' knowledge states and intentions.

Abstract

Cognitive anthropology suggests that the distinction of human intelligence lies in the ability to infer other individuals' knowledge states and understand their intentions. In comparison, our closest animal relative, chimpanzees, lack the capacity to do so. With this paper, we aim to evaluate LLM performance in the area of knowledge state tracking and estimation. We design two tasks to test (1) if LLMs can detect when story characters, through their actions, demonstrate knowledge they should not possess, and (2) if LLMs can predict story characters' next actions based on their own knowledge vs. objective truths they do not know. Results reveal that most current state-of-the-art LLMs achieve near-random performance on both tasks, and are substantially inferior to humans. We argue future LLM research should place more weight on the abilities of knowledge estimation and intention understanding.

Figures (11)

• Figure 1: The Knowledge-sensitive Next-action Prediction (KNP) task, where the implausible knowledge is highlighted for easy reading (but not highlighted for the LLMs and the human test-takers). This task tests whether LLMs can take the perspective of the story characters and choose actions based on knowledge they can access.
• Figure 2: Our dataset construction process (Section \ref{['sec: dataset']}). Using the original story, we first identify implausible knowledge that a character cannot possess at a given event, and that would affect the character's action. If no such knowledge exists, we prompt the model to introduce one. Then: 1) For the IKD task, we rewrite the original event to generate an erroneous story; 2) For the KNP task, we construct a question about a knowledge-sensitive action, creating two answer options—one without the implausible knowledge and one with the knowledge.
• Figure 3: Performance on different genres.
• Figure 4: Ablation: Focusing LLMs on Character Knowledge.
• Figure 5: Predictions on the IKD task (Erroneous Story). For easy reading, the logic error is in bold and implausible knowledge for the character is underlined.