Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO

Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, Jiaxing Huang

TL;DR

Share-GRPO introduces information sharing into reinforcement learning for multimodal large language models to enhance long-chain reasoning. By expanding the question space with semantically consistent transformations and sharing reasoning trajectories and rewards across variants, it mitigates sparse rewards and advantage vanishing. The approach integrates global and local hierarchical advantage estimation and shared policy optimization, achieving superior results on six reasoning benchmarks for both 7B and 32B MLLMs. Empirical findings show robust gains in mathematical and general reasoning tasks, with ablations confirming the effectiveness of each component and complementary gains when combined with dynamic sampling.

Abstract

In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over six widely-used reasoning benchmarks showcase the superior performance of our method. Code will be available at https://github.com/HJYao00/R1-ShareVL.

R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO

TL;DR

Share-GRPO introduces information sharing into reinforcement learning for multimodal large language models to enhance long-chain reasoning. By expanding the question space with semantically consistent transformations and sharing reasoning trajectories and rewards across variants, it mitigates sparse rewards and advantage vanishing. The approach integrates global and local hierarchical advantage estimation and shared policy optimization, achieving superior results on six reasoning benchmarks for both 7B and 32B MLLMs. Empirical findings show robust gains in mathematical and general reasoning tasks, with ablations confirming the effectiveness of each component and complementary gains when combined with dynamic sampling.

Abstract

In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over six widely-used reasoning benchmarks showcase the superior performance of our method. Code will be available at https://github.com/HJYao00/R1-ShareVL.

Paper Structure

This paper contains 18 sections, 8 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Model
  4. Reinforcement Learning for Multimodal Large Language Model Reasoning
  5. Information Sharing in Deep Learning
  6. Preliminary
  7. Share-GRPO
  8. Reasoning Space Expansion
  9. Shared Advantage Estimation
  10. Shared Policy Optimization
  11. Experiments
  12. Implementation Details
  13. Main Results
  14. Ablation Study
  15. Discussion
  16. ...and 3 more sections

Figures (3)

  • Figure 1: (a) Share-GRPO expands the question space via semantically consistent transformations, and then explores diverse reasoning trajectories from different question variants and shares the discovered trajectories among them. (b) Share-GRPO provides denser rewards and higher valid advantage ratios compared to GRPO, demonstrating its effectiveness in mitigating sparse reward and advantage vanishing issues. (c) Share-GRPO outperforms the baseline and other SOTA RL-based reasoning MLLMs on both mathematical and general reasoning benchmarks.
  • Figure 2: Overview of the proposed Share-GRPO. For a given question, Share-GRPO first applies semantically consistent transformation to generate a set of varied but semantically equivalent questions, thereby expanding the question space. It then encourages the MLLM to explore diverse reasoning paths over the expanded question space and facilitates the sharing of discovered reasoning trajectories and their rewards across the expanded questions during the reinforcement learning process.
  • Figure 3: Qualitative Results of reasoning capability of R1-ShareVL on mathematical problems.