Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Mental States to Guide Social Influence in Multi-Person Group Dialogue

Zhichao Liang, Satoshi Nakamura

TL;DR

SocialMindChange introduces a proactive Theory of Mind benchmark for multi-person group dialogue, moving beyond static state-tracking to planning dialogue that shifts a target's mental-state trajectory across five connected scenes with four participants. The framework combines a three-step construction pipeline (context, trajectories, scenario realization) with four evaluation question types to assess action guidance and cross-stage reasoning, including higher-order ToM. Large-scale experiments across nine diverse LLMs reveal a substantial gap to human performance, especially on higher-order, long-horizon tasks (Guidance--Transition), and show limited benefits from standard chain-of-thought prompting. The work provides a structured methodology for constructing, validating, and evaluating dynamic social reasoning and proactive guidance in LLMs, offering a clear direction for improving long-horizon, multi-agent mental-state reasoning in AI systems.

Abstract

Existing dynamic Theory of Mind (ToM) benchmarks mostly place language models in a passive role: the model reads a sequence of connected scenarios and reports what people believe, feel, intend, and do as these states change. In real social interaction, ToM is also used for action: a speaker plans what to say in order to shift another person's mental-state trajectory toward a goal. We introduce SocialMindChange, a benchmark that moves from tracking minds to changing minds in social interaction. Each instance defines a social context with 4 characters and five connected scenes. The model plays one character and generates dialogue across the five scenes to reach the target while remaining consistent with the evolving states of all participants. SocialMindChange also includes selected higher-order states. Using a structured four-step framework, we construct 1,200 social contexts, covering 6000 scenarios and over 90,000 questions, each validated for realism and quality. Evaluations on ten state-of-the-art LLMs show that their average performance is 54.2% below human performance. This gap suggests that current LLMs still struggle to maintain and change mental-state representations across long, linked interactions.

Understanding Mental States to Guide Social Influence in Multi-Person Group Dialogue

TL;DR

SocialMindChange introduces a proactive Theory of Mind benchmark for multi-person group dialogue, moving beyond static state-tracking to planning dialogue that shifts a target's mental-state trajectory across five connected scenes with four participants. The framework combines a three-step construction pipeline (context, trajectories, scenario realization) with four evaluation question types to assess action guidance and cross-stage reasoning, including higher-order ToM. Large-scale experiments across nine diverse LLMs reveal a substantial gap to human performance, especially on higher-order, long-horizon tasks (Guidance--Transition), and show limited benefits from standard chain-of-thought prompting. The work provides a structured methodology for constructing, validating, and evaluating dynamic social reasoning and proactive guidance in LLMs, offering a clear direction for improving long-horizon, multi-agent mental-state reasoning in AI systems.

Abstract

Existing dynamic Theory of Mind (ToM) benchmarks mostly place language models in a passive role: the model reads a sequence of connected scenarios and reports what people believe, feel, intend, and do as these states change. In real social interaction, ToM is also used for action: a speaker plans what to say in order to shift another person's mental-state trajectory toward a goal. We introduce SocialMindChange, a benchmark that moves from tracking minds to changing minds in social interaction. Each instance defines a social context with 4 characters and five connected scenes. The model plays one character and generates dialogue across the five scenes to reach the target while remaining consistent with the evolving states of all participants. SocialMindChange also includes selected higher-order states. Using a structured four-step framework, we construct 1,200 social contexts, covering 6000 scenarios and over 90,000 questions, each validated for realism and quality. Evaluations on ten state-of-the-art LLMs show that their average performance is 54.2% below human performance. This gap suggests that current LLMs still struggle to maintain and change mental-state representations across long, linked interactions.
Paper Structure (53 sections, 22 figures, 5 tables)

This paper contains 53 sections, 22 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theory of Mind Benchmarks for LLMs
  4. Group Interaction and Higher-Order Theory of Mind
  5. Human Behavior Simulation and Data Construction with LLMs
  6. SocialMindChange
  7. Building the SocialMindChange Benchmark: A Construction Framework
  8. Definitions and preliminaries.
  9. Construction pipeline.
  10. Step 1: Construction of Social Context
  11. Step 2: Design of Mental State Trajectories
  12. Step 3: Scenario Realization with Multi-party Dialogue and Labels
  13. Evaluation Question Types
  14. Guidance--Action (supportive next move under group constraints).
  15. Guidance--Transition-1 (immediate group state update).
  16. ...and 38 more sections

Figures (22)

  • Figure 1: An simplified illustration of how proactive guidance operates in a four-agent setting.
  • Figure 2: Benchmark Construction Framework & Evaluation Question Generation
  • Figure 3: Performance Breakdown of GPT-4o on ToM Tasks.
  • Figure 4: GPT-4o average score (scenarios 6–7)
  • Figure 5: Candidate Pool of Social Locations
  • ...and 17 more figures