Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning

TL;DR

SOPHIA introduces Semi-Off-Policy Reinforcement Learning to endow vision-language models with visual slow-thinking. By merging on-policy visual understanding with off-policy reasoning and propagating rewards from outcomes to visual cues, it mitigates misalignment and enables learning beyond current policies. Across eight multimodal benchmarks using InternVL base models, SOPHIA achieves state-of-the-art results among open-source LVLMs and competes with some closed-source systems on MathVision and OlympiadBench, while outperforming supervised fine-tuning and pure on-policy RL baselines. Limitations include long-range visual dependencies and potential hallucinations; future work targets stronger visual encoders, adaptive curricula, and broader data augmentation to further improve robustness and efficiency.

Abstract

Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking ability because the rollout space is restricted by its initial abilities. Off-policy RL offers a way to go beyond the current policy, but directly distilling trajectories from external models may cause visual hallucinations due to mismatched visual perception abilities across models. To address these issues, this paper proposes SOPHIA, a simple and scalable Semi-Off-Policy RL for vision-language slow-tHInking reAsoning. SOPHIA builds a semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off-policy slow-thinking reasoning from a language model, assigns outcome-based rewards to reasoning, and propagates visual rewards backward. Then LVLM learns slow-thinking reasoning ability from the obtained reasoning trajectories using propagated rewards via off-policy RL algorithms. Extensive experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50% in average, reaching state-of-the-art performance among open-source LVLMs on multiple multimodal reasoning benchmarks, and even outperforms some closed-source models (e.g., GPT-4.1) on the challenging MathVision and OlympiadBench, achieving 49.08% and 49.95% pass@1 accuracy, respectively. Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy RL methods, offering a better policy initialization for further on-policy training.

Figures (9)

• Figure 1: The average pass@1 accuracy of SOPHIA and some representative models across eight benchmarks, as well as their pass@1 accuracy on OlympiadBench, MathVision test, and MMMU Pro. InternVL3.0-38B + SOPHIA significantly improves over its base model, InternVL3.0-38B (Instruct), achieving state-of-the-art performance on most of the benchmarks.
• Figure 2: An overview of SOPHIA. SOPHIA samples reasoning trajectories $\bm{y}_i$ and calculates rewards for both reasoning $R(\bm{y}_i)$ and visual understanding $R(\bm{c}_i)$. Finally, it updates the LVLM policy via an off-policy method to cultivate visual slow-thinking reasoning abilities.
• Figure 3: The performance of InternVL3.0-38B and InternVL3.0-38B+SOPHIA on MMMU Pro and MathVision over 480 training steps, where the green and blue solid lines represent their respective pass@1 accuracy, while the dashed lines indicate the base model performance.
• Figure A1: Examples of different response styles. (a) is the QwQ Preview response style and (b) is the DeepSeek-R1 response style.
• Figure A2: Prompts for comprehensively captioning images.

Theorems & Definitions (2)

• Lemma A.1
• Lemma A.3