Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation

TL;DR

The paper identifies a gap in multimodal reasoning where vision is underutilized by MLLMs and demonstrates that simple caption-based baselines can outperform some vision-led models. It proposes a lightweight visual perturbation (VP) framework with three strategies—distractor concatenation, dominance-preserving mixup, and random rotation—that improve perceptual robustness without extra data or architectural changes. VP yields consistent 1–2 point gains across multiple math-visual benchmarks and training pipelines, and analysis reveals task-specific benefits and trade-offs among perturbations. The findings advocate for treating visual processing as a core component in multimodal reasoning and show VP as a practical, general-purpose enhancement.

Abstract

Despite the rapid progress of multimodal large language models (MLLMs), they have largely overlooked the importance of visual processing. In a simple yet revealing experiment, we interestingly find that language-only models, when provided with image captions, can achieve comparable or even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning. Motivated by this, we propose a simple visual perturbation framework that enhances perceptual robustness without requiring algorithmic modifications or additional training data. Our approach introduces three targeted perturbations: distractor concatenation, dominance-preserving mixup, and random rotation, that can be easily integrated into existing post-training pipelines including SFT, DPO, and GRPO. Through extensive experiments across multiple datasets, we demonstrate consistent improvements in mathematical reasoning performance, with gains comparable to those achieved through algorithmic changes. Additionally, we achieve competitive performance among open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual perturbation. Through comprehensive ablation studies, we analyze the effectiveness of different perturbation strategies, revealing that each perturbation type contributes uniquely to different aspects of visual reasoning. Our findings highlight the critical role of visual perturbation in multimodal mathematical reasoning: better reasoning begins with better seeing. Our code is available at https://github.com/YutingLi0606/Vision-Matters.

Figures (3)

• Figure 1: The left panel shows three settings: (i) Answer A (LLM-only), where a language model answers from text only; (ii) Answer B (MLLM), where a multimodal model jointly encodes both the question and the image; and (iii) Answer C (Caption-augmented LLM), where an image caption generated by the same MLLM is appended to the question for the LLM. The right panel presents quantitative results on MathVision wang2025mathvision. We interestingly find that language-only models, when provided with image captions, can sometimes achieve even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning.
• Figure 2: Our visual perturbation framework consists of three strategies: (1) distractor concatenation that horizontally combines the input image with a random distractor, (2) dominance-preserving mixup that blends the input with a distractor using skewed alpha values, and (3) random rotation that applies small angle rotations. During training, these perturbations are applied across multiple alignment pipelines including SFT, DPO, and GRPO to enhance the model's perceptual robustness and reasoning consistency.
• Figure 3: Visualization of different perturbation strategies used in our ablation studies. Specifically, Standard Mixup (0.45--0.55), blending two images with nearly equal weights; Gaussian Blur, implemented with a medium kernel radius (2.5--7.5) that degrades fine details while maintaining visibility; Gaussian Noise, adding pixel-level noise with standard deviation 50 or 150, which disrupts low-level visual signals.