From Sight to Insight: Improving Visual Reasoning Capabilities of Multimodal Models via Reinforcement Learning

TL;DR

This work identifies visual perception as the principal bottleneck limiting visual reasoning in multimodal LLMs solving visual puzzles. It demonstrates that substituting images with textual representations yields large performance gains, and investigates reward-driven reinforcement learning to elicit long, structured visual reasoning without dense annotations. Six reward functions, optimized via group relative policy optimization (GRPO), are evaluated on an expanded AlgoPuzzleVQA-based dataset, achieving a 5.56% improvement over the base Qwen-2.5-VL-7B model and demonstrating both in-domain and out-of-domain gains. The study reveals that carefully designed rewards can foster grounded, multi-step reasoning and generalize when trained on diverse data, while highlighting persistent challenges in perception and complex abstractions that warrant explicit visual grounding signals in future work.

Abstract

Reinforcement learning (RL) has emerged as a promising approach for eliciting reasoning chains before generating final answers. However, multimodal large language models (MLLMs) generate reasoning that lacks integration of visual information. This limits their ability to solve problems that demand accurate visual perception, such as visual puzzles. We show that visual perception is the key bottleneck in such tasks: converting images into textual descriptions significantly improves performance, yielding gains of 26.7% for Claude 3.5 and 23.6% for Claude 3.7. To address this, we investigate reward-driven RL as a mechanism to unlock long visual reasoning in open-source MLLMs without requiring costly supervision. We design and evaluate six reward functions targeting different reasoning aspects, including image understanding, thinking steps, and answer accuracy. Using group relative policy optimization (GRPO), our approach explicitly incentivizes longer, structured reasoning and mitigates bypassing of visual information. Experiments on Qwen-2.5-VL-7B achieve 5.56% improvements over the base model, with consistent gains across both in-domain and out-of-domain settings.

Figures (15)

• Figure 1: Overview of the GRPO-based training pipeline for MLLMs. For each sample, the policy model generates a completion conditioned on the prompt associated with a specific reward type (e.g., vanilla, mixture, continuous). Each completion is then evaluated using the reward calculation framework (Section \ref{['sec:reward-functions']}). Completions with different rewards are shown in Figure \ref{['completions']}. The resulting reward scores are passed into the GRPO trainer, which updates the policy model accordingly.
• Figure 2: Generic prompt structure used across all categories in both multimodal and text-only settings. In the text-only setting, the image is replaced with its textual representation so that the entire input is in text.
• Figure 3: An example comparing reasoning between the base (Qwen-2.5-VL-7B) and RL-trained versions with different reward strategies. With the Mixture reward, the model generates grounded, multi-step visual reasoning--accurately describing fine-grained image details such as the distinct colors of the hour and minute hands. Notably, this structured long-form reasoning emerges naturally from RL training (see Figures \ref{['reasoning-move-box']} and \ref{['reasoning-rotting-kiwi']} for additional examples). The model also performs step-by-step self-evaluation, revisiting and verifying each reasoning step before concluding with its final answer.
• Figure 4: Performance comparison across categories showing baseline vs. maximum scores and improvement margins. Reward functions that achieved the maximum score are indicated below each category. Results are reported for models trained on the Diverse-8K dataset.
• Figure 5: Question: Emily's event is going to start in 1 hour 10 minutes. The current time is shown on the clock. The clock is a standard analog clock without the second hand. What time will the event start?