Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu, Zhiqi Huang

TL;DR

MMErroR tackles the challenge of evaluating true multi-modal understanding by shifting from final-answer accuracy to process-level error detection and diagnosis. It introduces a 2,013-sample benchmark across 24 domains with a single injected error per chain, and defines two tasks: Error Type Classification (ETC) and Error Presence Detection (EPD). Across 20 vision-language models, even state-of-the-art systems achieve only 66.47% (ETC) and 61.25% (EPD), highlighting a significant gap in introspective reasoning. Through analyses of reasoning consistency, multi-modal alignment, and the impact of error awareness, the study identifies modality misalignment and limited reasoning verification as key bottlenecks, offering actionable directions for building more reliable, interpretable VLMs.

Abstract

Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models truly understand the content they process. Crucially, can VLMs detect when a reasoning process is wrong and identify its error type? To answer this, we present MMErroR, a multi-modal benchmark of 2,013 samples, each embedding a single coherent reasoning error. These samples span 24 subdomains across six top-level domains, ensuring broad coverage and taxonomic richness. Unlike existing benchmarks that focus on answer correctness, MMErroR targets a process-level, error-centric evaluation that requires models to detect incorrect reasoning and classify the error type within both visual and linguistic contexts. We evaluate 20 advanced VLMs, even the best model (Gemini-3.0-Pro) classifies the error in only 66.47\% of cases, underscoring the challenge of identifying erroneous reasoning. Furthermore, the ability to accurately identify errors offers valuable insights into the capabilities of multi-modal reasoning models. Project Page: https://mmerror-benchmark.github.io

MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

TL;DR

MMErroR tackles the challenge of evaluating true multi-modal understanding by shifting from final-answer accuracy to process-level error detection and diagnosis. It introduces a 2,013-sample benchmark across 24 domains with a single injected error per chain, and defines two tasks: Error Type Classification (ETC) and Error Presence Detection (EPD). Across 20 vision-language models, even state-of-the-art systems achieve only 66.47% (ETC) and 61.25% (EPD), highlighting a significant gap in introspective reasoning. Through analyses of reasoning consistency, multi-modal alignment, and the impact of error awareness, the study identifies modality misalignment and limited reasoning verification as key bottlenecks, offering actionable directions for building more reliable, interpretable VLMs.

Abstract

Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models truly understand the content they process. Crucially, can VLMs detect when a reasoning process is wrong and identify its error type? To answer this, we present MMErroR, a multi-modal benchmark of 2,013 samples, each embedding a single coherent reasoning error. These samples span 24 subdomains across six top-level domains, ensuring broad coverage and taxonomic richness. Unlike existing benchmarks that focus on answer correctness, MMErroR targets a process-level, error-centric evaluation that requires models to detect incorrect reasoning and classify the error type within both visual and linguistic contexts. We evaluate 20 advanced VLMs, even the best model (Gemini-3.0-Pro) classifies the error in only 66.47\% of cases, underscoring the challenge of identifying erroneous reasoning. Furthermore, the ability to accurately identify errors offers valuable insights into the capabilities of multi-modal reasoning models. Project Page: https://mmerror-benchmark.github.io
Paper Structure (30 sections, 1 equation, 6 figures, 5 tables)

This paper contains 30 sections, 1 equation, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. MMErroR
  3. Task Classification
  4. Error Type Classification (ETC)
  5. Error Presence Detection (EPD)
  6. Benchmark Construction
  7. Problem Curation
  8. Error Injection
  9. Data Verification
  10. Quality Assurance
  11. Data Analysis
  12. Experiment Settings
  13. Models
  14. Implementation and Metrics
  15. Empirical Results and Analysis
  16. ...and 15 more sections

Figures (6)

  • Figure 1: Comparison with existing error localization benchmarks. A sample from MMErroR illustrates an erroneous reasoning chain where the model is required to both detect and classify the error type.
  • Figure 2: Comparison of different VLMs across various task domains and four error types: Visual Perception Error (VPE), Reasoning Error (RE), Question Comprehension Error (QCE), and Knowledge Deployment Error (KDE).
  • Figure 3: Detailed analysis of domains, s and statistics of MMErroR.
  • Figure 4: Performance comparison of different VLMs on MMErroR. We evaluate and compare performance under two settings: Error Type Classification (ETC) and Error Presence Detection (EPD).
  • Figure 5: Visualization of the logit lens for image tokens.
  • ...and 1 more figures