Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning

TL;DR

This work argues that deep affective understanding in multimodal LLMs requires Theory of Mind (ToM) and introduces HitEmotion, a hierarchical ToM-based benchmark spanning three cognitive depths across 24 datasets to diagnose emotional reasoning capabilities. It also proposes TMPO, a ToM-guided reasoning framework that combines prompting, supervised fine-tuning, and Group-wise Reward Policy Optimization (GRPO) to shape and evaluate intermediate mental states as supervision signals. HitEmotion reveals pronounced deficits in current models at higher cognitive depths, while ToM prompting and especially TMPO yield substantial gains in end-task accuracy and the faithfulness and coherence of generated rationales, sometimes outperforming leading proprietary systems on cognitively demanding tasks. Collectively, HitEmotion and TMPO offer a practical toolkit for evaluating and advancing cognition-driven emotional understanding in multimodal LLMs, moving beyond surface-level emotion recognition toward robust mental-state simulation. The authors also provide datasets and code to enable reproducibility and broader adoption.

Abstract

Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs. Our dataset and code are available at: https://HitEmotion.github.io/.

Figures (80)

• Figure 1: Overview of our HitEmotion benchmark.
• Figure 2: Task taxonomy and examples in our HitEmotion benchmark.
• Figure 3: Our reasoning chain curation pipeline.
• Figure 4: Average performance across our HitEmotion benchmark levels. Comparison of 17 multimodal models on our HitEmotion benchmark, showing average scores for each level per model.
• Figure 5: Ablation study on ToM-style prompting.