Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models

TL;DR

PercepToM is presented, a novel ToM method leveraging LLMs’ strong perception inference capability while supplementing their limited perception-to-belief inference, and demonstrates that PercepToM significantly enhances LLM’s performance, especially in false belief scenarios.

Abstract

While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs' ToM abilities by evaluating key human ToM precursors$-$perception inference and perception-to-belief inference$-$in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions on ToMi and FANToM, respectively. Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control). Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM's performance, especially in false belief scenarios.

Figures (5)

• Figure 1: Inspired by children's developmental trajectory for theory of mind (ToM), our perception-augmented ToM benchmarks test the two precursory inferences of ToM in LLMs in order to examine their underlying social reasoning capabilities: (1) perception inference and (2) perception-to-belief inference (§ \ref{['sec:method_benchmark']}).
• Figure 2: Example data in Percept-ToMi and Percept-FANToM. For each context, the perceivers of every scene description or utterance are annotated automatically (Percept-ToMi) and manually (Percept-FANToM).
• Figure 3: Perception inference, perception-to-belief inference, and ToM performances of LLMs in true and false belief scenarios of Percept-ToMi and Percept-FANToM. Although the models exhibit similar accuracy in perception inference across both scenarios, their performance in perception-to-belief inference and ToM scenarios varies significantly.
• Figure 4: Pearson correlation of LLMs' ToM performance with perception inference (left) and perception-to-belief inference (right) performances. ToM performance shows a positive correlation with perception-to-belief inference performance but exhibits a weak or no correlation with perception inference performance.
• Figure 5: An overview of our PercepToM framework, which enhances LLMs' ToM reasoning by (1) instructing LLMs to infer the perceivers of each information in the context; (2) aiding their perception-to-belief inference through the perspective context extraction step, which isolates the context perceived by the target character; and (3) allowing LLMs to generate responses to ToM questions based on this perspective context.