Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-THEBench: Do Reasoning MLLMs Think Reasonably?

Zhidian Huang, Zijun Yao, Ji Qi, Shangqing Tu, Junxian Ma, Jinxin Liu, Weichuan Liu, Xiaoyin Che, Lei Hou, Juanzi Li

TL;DR

MM-THEBench provides a structured, automated framework to evaluate hallucinations in intermediate CoTs produced by reasoning MLLMs across multimodal tasks. It introduces a two-layer, fine-grained taxonomy anchored in knowledge, perception, and reasoning, and a three-level evaluation pipeline (answer, step, rubric) powered by an LLM-as-judge. Empirical results across 14 mainstream models reveal that intermediate CoTs often lag behind final answers in fidelity, with perception hallucinations being common yet less harmful than reasoning hallucinations that strongly relate to incorrect outcomes, especially in spatial reasoning. The benchmark enables nuanced monitoring of thinking quality and highlights the need for monitoring and mitigating thinking hallucinations to improve the reliability of multimodal reasoning systems.

Abstract

Recent advances in multimodal large language models (MLLMs) mark a shift from non-thinking models to post-trained reasoning models capable of solving complex problems through thinking. However, whether such thinking mitigates hallucinations in multimodal perception and reasoning remains unclear. Self-reflective reasoning enhances robustness but introduces additional hallucinations, and subtle perceptual errors still result in incorrect or coincidentally correct answers. Existing benchmarks primarily focus on models before the emergence of reasoning MLLMs, neglecting the internal thinking process and failing to measure the hallucinations that occur during thinking. To address these challenges, we introduce MM-THEBench, a comprehensive benchmark for assessing hallucinations of intermediate CoTs in reasoning MLLMs. MM-THEBench features a fine-grained taxonomy grounded in cognitive dimensions, diverse data with verified reasoning annotations, and a multi-level automated evaluation framework. Extensive experiments on mainstream reasoning MLLMs reveal insights into how thinking affects hallucination and reasoning capability in various multimodal tasks.

MM-THEBench: Do Reasoning MLLMs Think Reasonably?

TL;DR

MM-THEBench provides a structured, automated framework to evaluate hallucinations in intermediate CoTs produced by reasoning MLLMs across multimodal tasks. It introduces a two-layer, fine-grained taxonomy anchored in knowledge, perception, and reasoning, and a three-level evaluation pipeline (answer, step, rubric) powered by an LLM-as-judge. Empirical results across 14 mainstream models reveal that intermediate CoTs often lag behind final answers in fidelity, with perception hallucinations being common yet less harmful than reasoning hallucinations that strongly relate to incorrect outcomes, especially in spatial reasoning. The benchmark enables nuanced monitoring of thinking quality and highlights the need for monitoring and mitigating thinking hallucinations to improve the reliability of multimodal reasoning systems.

Abstract

Recent advances in multimodal large language models (MLLMs) mark a shift from non-thinking models to post-trained reasoning models capable of solving complex problems through thinking. However, whether such thinking mitigates hallucinations in multimodal perception and reasoning remains unclear. Self-reflective reasoning enhances robustness but introduces additional hallucinations, and subtle perceptual errors still result in incorrect or coincidentally correct answers. Existing benchmarks primarily focus on models before the emergence of reasoning MLLMs, neglecting the internal thinking process and failing to measure the hallucinations that occur during thinking. To address these challenges, we introduce MM-THEBench, a comprehensive benchmark for assessing hallucinations of intermediate CoTs in reasoning MLLMs. MM-THEBench features a fine-grained taxonomy grounded in cognitive dimensions, diverse data with verified reasoning annotations, and a multi-level automated evaluation framework. Extensive experiments on mainstream reasoning MLLMs reveal insights into how thinking affects hallucination and reasoning capability in various multimodal tasks.
Paper Structure (33 sections, 1 equation, 11 figures, 9 tables)

This paper contains 33 sections, 1 equation, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Proposed MM-THEBench
  3. Benchmark Design
  4. Data Collection
  5. Data Annotation and Quality Control
  6. Data Statistics and Analysis
  7. The Proposed Evaluation Framework
  8. Judge Model
  9. Evaluation Pipeline
  10. Evaluation Metrics
  11. Experiments
  12. Experimental Setup
  13. Overall Performance on Image-Based Data
  14. Overall Performance on Video-Based Data
  15. Hallucination Analysis
  16. ...and 18 more sections

Figures (11)

  • Figure 1: Overview of the evaluation framework in MM-THEBench. In (c) Rubric-level Evaluation, the column "RID" denotes the rubric item ID. "Judge" indicates whether the judge model considers the item satisfied in the intermediate CoT, and "Hallu" indicates whether a hallucination occurs for the corresponding rubric item. "K", "P", "R", represents the dimensions Knowledge, Perception, and Reasoning. "H-score" denotes Hallucination-free score (1-hallucination score ratio) .
  • Figure 2: Data composition in MM-THEBench. Left: Modality composition. Right: Question format composition.
  • Figure 3: Hallucination category distribution by model and answer correctness. Qwen3-VL-8B denotes Qwen3-VL-8B-Thinking, and Qwen3-VL-235B denotes Qwen3-VL-235B-A22B-Thinking.
  • Figure 4: Distribution of hallucination subcategories. Segment sizes correspond to cumulative scores.
  • Figure 5: Relationship between thinking length and quality, where the X-axis represents the relative length of thinking tokens, Y-axis represents step-level precision (left) and recall (right).
  • ...and 6 more figures