Grounded Reinforcement Learning for Visual Reasoning

TL;DR

ViGoRL introduces a visually grounded reinforcement learning paradigm that anchors each reasoning step to explicit image coordinates, addressing the tendency of vision-language models to produce ungrounded reasoning. By combining MCTS-generated, spatially grounded traces (warm-start) with GRPO-based RL and a multi-turn visual feedback loop that enables zoomed-in inspections, ViGoRL achieves substantial gains across spatial reasoning, GUI grounding, and live visual web tasks. Ablation studies show explicit grounding, MCTS warm-start, and RL refinement are all critical to performance and to amplifying targeted visual reasoning behaviors such as region exploration, subgoal setting, and verification. Human evaluations confirm that the grounding is often accurate and helpful for understanding the model’s reasoning, suggesting grounding as a powerful cognitive scaffold for general visual reasoning with broad implications for interpretability and generalization.

Abstract

While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.

Figures (7)

• Figure 1: Grounded visual reasoning enables interpretable and accurate answers. ViGoRL decomposes the task into a sequence of natural language thoughts anchored in image regions. In contrast, Vanilla GRPO and SFT baselines produce ungrounded and incorrect responses.
• Figure 2: Without actively reinforcing visually grounded behaviors, RL collapses onto shortcuts that maximize immediate rewards at the expense of richer visual reasoning. Standard CoT and Vanilla GRPO (left and center) exhibit visually ungrounded reasoning, relying on vague references to scene elements (shown in yellow), which often results in incorrect answers (marked in red). In contrast, Visually Grounded RL (right) explicitly references object positions, demonstrating precise spatial grounding (shown in blue) and more often producing correct reasoning outcomes (marked in green). See Section \ref{['sec:behavioral_results']} for further analysis.
• Figure 3: Overview of the ViGoRL approach. (Left) We use MCTS with a teacher model to generate reasoning chains grounded in specific image regions. (Middle) These reasoning trees are linearized and used for supervised fine-tuning (SFT) to train a base model. (Right) We apply GRPO with an outcome-based reward to further refine the grounded reasoning.
• Figure 4: Human evaluation of grounded reasoning. Participants judged the grounded predictions as both accurate and helpful when correct.
• Figure A1: Response length with and without turn bonus. Without the bonus, the model converges to always taking a single turn (as also verified by examining model outputs), whereas the bonus enables the model to stabilize multi-turn.