Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models

Haibo Tong, Zeyang Yue, Feifei Zhao, Erliang Lin, Lu Jia, Ruolin Chen, Yinqian Sun, Qian Zhang, Yi Zeng

TL;DR

CogToM addresses the fragmented evaluation of Theory of Mind (ToM) in large language models by introducing a theory-grounded benchmark that blends psychological paradigms into 46 scene-based, bilingual ToM tasks exceeding 8,000 instances. The dataset is built through a six-stage pipeline combining data collection, LLM-based expansion, and rigorous human annotation (with 49 experts) to ensure high-quality, diverse coverage mapped to 36 sub-capabilities across 7 cognitive dimensions under the ATOMS framework. A broad evaluation of 22 models, including frontier systems like GPT-5.1 and Qwen3-Max, reveals strong heterogeneity across ToM dimensions and significant discriminative power from the newly introduced tasks, with evidence of Moravec’s paradox-like patterns in model cognition. CogToM thus offers a robust instrument for probing the cognitive boundaries of LLM ToM and informs future model development and evaluation strategies in cross-lingual, theory-driven contexts.

Abstract

Whether Large Language Models (LLMs) truly possess human-like Theory of Mind (ToM) capabilities has garnered increasing attention. However, existing benchmarks remain largely restricted to narrow paradigms like false belief tasks, failing to capture the full spectrum of human cognitive mechanisms. We introduce CogToM, a comprehensive, theoretically grounded benchmark comprising over 8000 bilingual instances across 46 paradigms, validated by 49 human annotator.A systematic evaluation of 22 representative models, including frontier models like GPT-5.1 and Qwen3-Max, reveals significant performance heterogeneities and highlights persistent bottlenecks in specific dimensions. Further analysis based on human cognitive patterns suggests potential divergences between LLM and human cognitive structures. CogToM offers a robust instrument and perspective for investigating the evolving cognitive boundaries of LLMs.

CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models

TL;DR

CogToM addresses the fragmented evaluation of Theory of Mind (ToM) in large language models by introducing a theory-grounded benchmark that blends psychological paradigms into 46 scene-based, bilingual ToM tasks exceeding 8,000 instances. The dataset is built through a six-stage pipeline combining data collection, LLM-based expansion, and rigorous human annotation (with 49 experts) to ensure high-quality, diverse coverage mapped to 36 sub-capabilities across 7 cognitive dimensions under the ATOMS framework. A broad evaluation of 22 models, including frontier systems like GPT-5.1 and Qwen3-Max, reveals strong heterogeneity across ToM dimensions and significant discriminative power from the newly introduced tasks, with evidence of Moravec’s paradox-like patterns in model cognition. CogToM thus offers a robust instrument for probing the cognitive boundaries of LLM ToM and informs future model development and evaluation strategies in cross-lingual, theory-driven contexts.

Abstract

Whether Large Language Models (LLMs) truly possess human-like Theory of Mind (ToM) capabilities has garnered increasing attention. However, existing benchmarks remain largely restricted to narrow paradigms like false belief tasks, failing to capture the full spectrum of human cognitive mechanisms. We introduce CogToM, a comprehensive, theoretically grounded benchmark comprising over 8000 bilingual instances across 46 paradigms, validated by 49 human annotator.A systematic evaluation of 22 representative models, including frontier models like GPT-5.1 and Qwen3-Max, reveals significant performance heterogeneities and highlights persistent bottlenecks in specific dimensions. Further analysis based on human cognitive patterns suggests potential divergences between LLM and human cognitive structures. CogToM offers a robust instrument and perspective for investigating the evolving cognitive boundaries of LLMs.
Paper Structure (32 sections, 11 figures, 52 tables)

This paper contains 32 sections, 11 figures, 52 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. CogToM Framework
  4. Task Design
  5. Construction of the CogToM Dataset
  6. Data Collection
  7. LLM-based Question Expansion
  8. Human Annotation and Data Validation
  9. Experimental Results and Analysis
  10. Experimental Setups
  11. Evaluated Models
  12. Evaluation Methods and Metrics
  13. Main Results
  14. Temporal Evolution and Scaling Dynamics
  15. Differences Across ToM Cognitive Dimensions
  16. ...and 17 more sections

Figures (11)

  • Figure 1: Comparison of task coverage across different ToM benchmarks.
  • Figure 2: Overview of the CogToM framework.
  • Figure 3: The data construction pipeline of CogToM.
  • Figure 4: Overall average performance of open-source and closed-source models across various release dates, scales and families.
  • Figure 5: Accuracy distribution of models across different ToM task categories.
  • ...and 6 more figures