Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters

Chulun Zhou, Qiujing Wang, Mo Yu, Xiaoqian Yue, Rui Lu, Jiangnan Li, Yifan Zhou, Shunchi Zhang, Jie Zhou, Wai Lam

TL;DR

This paper verifies the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios, and introduces CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels.

Abstract

Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global context, especially personal background of characters. In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios. To achieve this, we introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 and DeepSeek-R1 models, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.

The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters

TL;DR

This paper verifies the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios, and introduces CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels.

Abstract

Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global context, especially personal background of characters. In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios. To achieve this, we introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 and DeepSeek-R1 models, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.
Paper Structure (38 sections, 1 equation, 4 figures, 19 tables)

This paper contains 38 sections, 1 equation, 4 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation of ToM in Global Contexts
  4. Problem Formulation
  5. ToM Dimensions Studied
  6. CharToM-QA: Assessing the Contextual Aspect of ToM
  7. Book Notes Collection & Filtering
  8. Construction of Generative QA Task
  9. Key Note Extraction & Paraphrasing (i.e., Answer Generation).
  10. Question Generation & Verification.
  11. Construction of Multichoice QA Task
  12. Evaluation Protocols for Generative QA
  13. Experiments
  14. Setup
  15. Evaluated LLMs
  16. ...and 23 more sections

Figures (4)

  • Figure 1: An example from CharToM-QA benchmark. LLMs make their responses given a question and corresponding story plot in generative and multichoice QA settings.
  • Figure 2: Illustration of notes from users on reading apps. A user underlines a text fragment (red) and writes his note about the character "Mrs. Bennet".
  • Figure 3: The extract-then-paraphrase approach to fetch ToM descriptions of a target character. The highlighted text pieces are the key notes extracted by annotators, which are then paraphrased into complete statements.
  • Figure 4: Given the corresponding story plot, GPT-4o is used to generate candidate questions about the target character using ToM descriptions. For each dimension, at most 4 candidates are kept after verification. Then, annotators manually choose the best one from these candidates.