Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward

Jing Bi, Guangyu Sun, Ali Vosoughi, Chen Chen, Chenliang Xu

TL;DR

The paper tackles visual reasoning challenges in multimodal LLMs, focusing on hallucinations and reliance on textual priors. It introduces a three-stage diagnostic framework and an agent-based architecture that couples LLM reasoning with lightweight visual modules to enable iterative refinement of reasoning chains. Across MMMU and MathVista, the approach yields substantial gains over a 7B baseline, matching or surpassing larger models by leveraging perceptual grounding with tools like OCR, captioning, and a Python interpreter. By releasing its framework and evaluation suite, it provides a practical path for diagnosing and improving visual reasoning in real-world settings.

Abstract

Multimodal large language models (MLLMs) that integrate visual and textual reasoning leverage chain-of-thought (CoT) prompting to tackle complex visual tasks, yet continue to exhibit visual hallucinations and an over-reliance on textual priors. We present a systematic diagnosis of state-of-the-art vision-language models using a three-stage evaluation framework, uncovering key failure modes. To address these, we propose an agent-based architecture that combines LLM reasoning with lightweight visual modules, enabling fine-grained analysis and iterative refinement of reasoning chains. Our results highlight future visual reasoning models should focus on integrating a broader set of specialized tools for analyzing visual content. Our system achieves significant gains (+10.3 on MMMU, +6.0 on MathVista over a 7B baseline), matching or surpassing much larger models. We will release our framework and evaluation suite to facilitate future research.

Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward

TL;DR

The paper tackles visual reasoning challenges in multimodal LLMs, focusing on hallucinations and reliance on textual priors. It introduces a three-stage diagnostic framework and an agent-based architecture that couples LLM reasoning with lightweight visual modules to enable iterative refinement of reasoning chains. Across MMMU and MathVista, the approach yields substantial gains over a 7B baseline, matching or surpassing larger models by leveraging perceptual grounding with tools like OCR, captioning, and a Python interpreter. By releasing its framework and evaluation suite, it provides a practical path for diagnosing and improving visual reasoning in real-world settings.

Abstract

Multimodal large language models (MLLMs) that integrate visual and textual reasoning leverage chain-of-thought (CoT) prompting to tackle complex visual tasks, yet continue to exhibit visual hallucinations and an over-reliance on textual priors. We present a systematic diagnosis of state-of-the-art vision-language models using a three-stage evaluation framework, uncovering key failure modes. To address these, we propose an agent-based architecture that combines LLM reasoning with lightweight visual modules, enabling fine-grained analysis and iterative refinement of reasoning chains. Our results highlight future visual reasoning models should focus on integrating a broader set of specialized tools for analyzing visual content. Our system achieves significant gains (+10.3 on MMMU, +6.0 on MathVista over a 7B baseline), matching or surpassing much larger models. We will release our framework and evaluation suite to facilitate future research.
Paper Structure (7 sections, 4 figures, 4 tables)

This paper contains 7 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Diagnostic Methodology
  3. Comparative Model Analysis
  4. In-depth Examination of QVQ
  5. Agent-based Diagnostic and Intervention
  6. Ablation Study
  7. Conclusion and Limitations

Figures (4)

  • Figure 1: This image showcases our agent system that leverages a pure LLM to solve a visual reasoning problem using external tools. It illustrates how complex tasks, such as fitting traffic speed-density data to the Greenshields model, can offload substantial token usage to a code interpreter, highlighting an efficient division of labor between perception, reasoning and computation.
  • Figure 2: Comparison of token utilization and accuracy between Mulberry-7B and LLaVA-CoT-11B. Both models exhibit less adaptive reasoning, with token usage. We observe that LLaVA-CoT-11B frequently generates more tokens than Mulberry-7B, while Mulberry-7B tends to centralize its token usage around 300--350 tokens.
  • Figure 3: Token usage and accuracy trends for QVQ on MMMU. Left: Accuracy as a function of token count, showing diminishing returns and a decline beyond 2,000 tokens. Right: Distribution of token counts for correct and incorrect answers, including unfinished answers
  • Figure 4: This chart illustrates the number of correct and incorrect cases across different token count ranges, with a green line as accuracy ratio. As token count increases, the number of cases generally decreases, and the accuracy ratio tends to decline.