More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

TL;DR

The paper reveals a dual nature of multimodal reasoning in vision-language models: while extended reasoning enhances logical inference, it can undermine perceptual grounding through visual forgetting. It identifies this forgetting via attention dynamics and error analysis, and demonstrates that simple inference-time remedies are insufficient. The authors propose Vision-Anchored Policy Optimization (VAPO), a training-based RL approach that introduces visual anchors and a perception reward to steer reasoning along visually grounded trajectories, yielding state-of-the-art results on ten benchmarks with VAPO-Thinker-7B. The work highlights the importance of grounding multimodal reasoning and provides a practical, efficient method to improve perceptual reliance and reliability in VLMs.

Abstract

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/

Figures (15)

• Figure 1: The examples where more reasoning yields less accuracy. We select Vision-R1 huang2025vision as a representitive multimodal reasoning model and evaluate on established benchmarks. Given the ground truth (GT) answer, we examine three settings, i.e., direct (DT) answering, full reasoning, and early decision. Correct and incorrect responses are highlighted in green and red, respectively, while designed prompts for early decision are indicated in blue.
• Figure 2: The dual nature analysis of reasoning. In (A), we depict how accuracy evolves throughout the reasoning process. In (B), we show the distribution of error categories under full reasoning (inner ring), alongside the proportion recoverable via early decision (outer ring). In (C), we further present the initial ratio of perception errors on different benchmarks (left bar), as well as the remaining errors that persist after recovery (right bar). It is important to note that recoverable ratio is a loose metric based on a series of early decisions along the reasoning trajectory rather than a single evaluation.
• Figure 3: The analysis of visual forgetting. In (A), we randomly select an example from MathVerse and visualize the evolution of attention ratio to visual tokens from Vision-R1 across generation steps. We consider vanilla reasoning as well as two variants, i.e., visual replay and focus prompt. In (B), we report the average accuracy across established benchmarks and models, and track its trend throughout the reasoning process under the aforementioned three settings. The average cutoff positions across examples to insert images or instructions are indicated in vertical dash lines.
• Figure 4: The overview of VAPO. On the left, we first employ GPT to generate a set of claims about visual input, each of which may be either factually correct or not. These claims are then treated as anchors and inserted into the model's reasoning process. Specifically, on the right, for each anchor, we randomly sample a prefix from the reasoning trajectory, append the claim to this truncated context, and probe the model judgment regarding the claim's validity.
• Figure 5: The evaluation of our proposed method on general-purpose benchmarks. All baselines considered in this evaluation are of the 7B scale.