Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models

TL;DR

This paper questions the emergence of Theory of Mind (ToM) in large language models by using a Generative Agent-Based Model (Concordia) to embed utterances in realistic social environments. It evaluates ToM as the ability to infer intended meanings from extralinguistic context and map those inferences to appropriate actions, using a two-phase experimental setup with MCQA action selection and subsequent coherence assessment. Across 200 simulations and five task variants, GPT-4o-mini shows limited ToM-like behavior: actions rarely align with belief attributions, and generated causal effects lack coherence with the social context, suggesting that observed ToM in prior work may arise from statistical memorization rather than true reasoning. The study underscores the need for action-based, context-sensitive evaluation frameworks to robustly assess social reasoning in LLMs and cautions against overinterpreting emergent ToM capabilities.

Abstract

Language is fundamental to human cooperation, facilitating not only the exchange of information but also the coordination of actions through shared interpretations of situational contexts. This study explores whether the Generative Agent-Based Model (GABM) Concordia can effectively model Theory of Mind (ToM) within simulated real-world environments. Specifically, we assess whether this framework successfully simulates ToM abilities and whether GPT-4 can perform tasks by making genuine inferences from social context, rather than relying on linguistic memorization. Our findings reveal a critical limitation: GPT-4 frequently fails to select actions based on belief attribution, suggesting that apparent ToM-like abilities observed in previous studies may stem from shallow statistical associations rather than true reasoning. Additionally, the model struggles to generate coherent causal effects from agent actions, exposing difficulties in processing complex social interactions. These results challenge current statements about emergent ToM-like capabilities in LLMs and highlight the need for more rigorous, action-based evaluation frameworks.

Figures (8)

• Figure 1: The Game Master mediates between agents and the environment, translating agent actions into environmental observations, while agents adapt their actions based on memory and updated observations.
• Figure 2: An example of the agents' memory and the type of observation, where the stimulus reproduces an Indirect Request.
• Figure 3: Adjustment of agents' knowledge for the same stimulus across different task designs.Tasks 4 and 5 involve second-order beliefs. The attempted action of the listener agent is determined by the interpretation of the final utterance, which may be understood either literally or non-literally. The action that aligns exclusively with the literal interpretation is highlighted in yellow.
• Figure 4: Percentage of correct answers for each task. Orange bars represent the listener, whose correct response varies depending on the scenario. IR: Indirect Requests; IS: Indirect Suggestions; ID: Indirect Declinations; IR-Os: Indirect Requests extracted from trott-bergen; IT: Indirect Threats; VI-IH: Verbal Irony, Indirect Hyperbole; VI-IQ: Verbal Irony, Rhetorical Questions; VI-IS: Verbal Irony, Sarcasm
• Figure 5: Ratings are assigned by prompting the model to evaluate the coherence of the generated effects on agents in relation to the given context and attempted actions. When no effect on the agents is present, the model must assign a rating of 0 -- the probability of such cases is displayed in the box at the top right. For all other instances where an effect is generated, the assigned coherence ratings for both the speaker and the listener must fall within the range of 1 to 5.