Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

TL;DR

<3-5 sentence high-level summary> Visionary-R1 tackles the shortcut problem in reinforcement-learning-based visual reasoning by enforcing image understanding before reasoning through a caption–reason–answer pipeline trained with a caption reward. Trained solely on CoT-free QA pairs (272.6K), Visionary-R1 demonstrates strong generalization across diverse visual reasoning benchmarks, outperforming large proprietary multimodal systems like GPT-4o and Claude3.5-Sonnet on several tasks. The approach highlights the necessity of grounding visual input via captioning and shows that a dynamically annealed KL penalty stabilizes training and enables longer, more informative reasoning. This work suggests that RL for visual reasoning can reach strong performance without chain-of-thought supervision, with potential benefits from scaling to larger models.</p>

Abstract

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

Figures (14)

• Figure 1: Comparison between the GRPO model and Visionary-R1. Using the reason-answer output format, the GRPO model tends to generate shortcut responses for easy samples during training, which hinders the model from learning general-purpose reasoning capabilities and results in poor generalization performance. In contrast, with a more comprehensive understanding of the image context, i.e., using the caption-reason-answer output format, Visionary-R1 consistently generates long, meaningful reasoning chains for both easy and hard samples.
• Figure 2: The longer the reasoning chain, the better the accuracy.
• Figure 3: Overview of Visionary-R1. The primary training pipeline utilizes the GRPO method, which generates multiple reasoning paths for each question-answer pair. Additionally, an info tag is incorporated when calculating the format reward, and the policy model's LLM part is used to answer questions based on the description between the info tags, serving as the caption rewards. All rewards are then aggregated to determine the final advantage of each path.
• Figure 4: Visualization of different model outputs. The caption output format enhances the reasoning while the caption reward further makes the reasoning more in-depth by improving the caption quality.
• Figure 5: Visualization of curves for different KL coefficients (top) and output examples (bottom).