Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning

Zirun Guo, Minjie Hong, Tao Jin

TL;DR

Observe-R1 presents a progressive RL framework to enhance reasoning in multimodal LLMs, leveraging the NeuraLadder curriculum, a structured multimodal observation format, a bonus reward, and dynamic sampling to train more concise and clear reasoning. It demonstrates that a 3B model, when trained with NeuraLadder and the proposed constraints, can outperform several larger reasoning models on math and science benchmarks while maintaining general multimodal performance. Ablation studies validate the contribution of each component, and the authors provide open-source NeuraLadder data and code for reproducibility and further research. This approach offers a scalable path to improved multimodal reasoning without resorting to significantly larger models.

Abstract

Reinforcement Learning (RL) has shown promise in improving the reasoning abilities of Large Language Models (LLMs). However, the specific challenges of adapting RL to multimodal data and formats remain relatively unexplored. In this work, we present Observe-R1, a novel framework aimed at enhancing the reasoning capabilities of multimodal large language models (MLLMs). We draw inspirations from human learning progression--from simple to complex and easy to difficult, and propose a gradual learning paradigm for MLLMs. To this end, we construct the NeuraLadder dataset, which is organized and sampled according to the difficulty and complexity of data samples for RL training. To tackle multimodal tasks, we introduce a multimodal format constraint that encourages careful observation of images, resulting in enhanced visual abilities and clearer and more structured responses. Additionally, we implement a bonus reward system that favors concise, correct answers within a length constraint, alongside a dynamic weighting mechanism that prioritizes uncertain and medium-difficulty problems, ensuring that more informative samples have a greater impact on training. Our experiments with the Qwen2.5-VL-3B and Qwen2.5-VL-7B models on 20k samples from the NeuraLadder dataset show that Observe-R1 outperforms a series of larger reasoning models on both reasoning and general benchmarks, achieving superior clarity and conciseness in reasoning chains. Ablation studies validate the effectiveness of our strategies, highlighting the robustness and generalization of our approach. The dataset and code will be released at https://github.com/zrguo/Observe-R1.

Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning

TL;DR

Observe-R1 presents a progressive RL framework to enhance reasoning in multimodal LLMs, leveraging the NeuraLadder curriculum, a structured multimodal observation format, a bonus reward, and dynamic sampling to train more concise and clear reasoning. It demonstrates that a 3B model, when trained with NeuraLadder and the proposed constraints, can outperform several larger reasoning models on math and science benchmarks while maintaining general multimodal performance. Ablation studies validate the contribution of each component, and the authors provide open-source NeuraLadder data and code for reproducibility and further research. This approach offers a scalable path to improved multimodal reasoning without resorting to significantly larger models.

Abstract

Reinforcement Learning (RL) has shown promise in improving the reasoning abilities of Large Language Models (LLMs). However, the specific challenges of adapting RL to multimodal data and formats remain relatively unexplored. In this work, we present Observe-R1, a novel framework aimed at enhancing the reasoning capabilities of multimodal large language models (MLLMs). We draw inspirations from human learning progression--from simple to complex and easy to difficult, and propose a gradual learning paradigm for MLLMs. To this end, we construct the NeuraLadder dataset, which is organized and sampled according to the difficulty and complexity of data samples for RL training. To tackle multimodal tasks, we introduce a multimodal format constraint that encourages careful observation of images, resulting in enhanced visual abilities and clearer and more structured responses. Additionally, we implement a bonus reward system that favors concise, correct answers within a length constraint, alongside a dynamic weighting mechanism that prioritizes uncertain and medium-difficulty problems, ensuring that more informative samples have a greater impact on training. Our experiments with the Qwen2.5-VL-3B and Qwen2.5-VL-7B models on 20k samples from the NeuraLadder dataset show that Observe-R1 outperforms a series of larger reasoning models on both reasoning and general benchmarks, achieving superior clarity and conciseness in reasoning chains. Ablation studies validate the effectiveness of our strategies, highlighting the robustness and generalization of our approach. The dataset and code will be released at https://github.com/zrguo/Observe-R1.
Paper Structure (16 sections, 7 equations, 7 figures, 2 tables)

This paper contains 16 sections, 7 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning in Large Models
  4. Reinforcement Learning for Large Models
  5. Methodology
  6. Preliminaries
  7. NeuraLadder Dataset
  8. Observe-R1
  9. Multimodal Format
  10. Additional Bonus Reward
  11. Dynamic Weighting and Sampling
  12. Experiments
  13. Implementation Details
  14. Main Results
  15. Ablation Study
  16. ...and 1 more sections

Figures (7)

  • Figure 1: (a) NeuraLadder construction pipeline. (b) Proportion, average complexity and complexity of the correct responses of the NeuraLadder dataset. (c) Sampled and Mixed results of the dataset.
  • Figure 2: A response example from Observe-R1.
  • Figure 3: An example from MathVista. Text with a grey background highlights the contradictions between the two sentences. The GRPO baseline accurately observes the image but fails to retain its content during reasoning.
  • Figure 4: A visual-only problem example from MathVerse. The GRPO baseline fails to observe the image accurately.
  • Figure 5: (a) Total rewards on NeuraLadder and randomly sampled dataset. (b) Response length on NeuraLadder and randomly sampled dataset. (c) Total rewards with multimodal format. (d) Response length with multimodal format. We get these results using Qwen2.5-VL-3B on the same set of data. If the answer and format are both correct, the total reward will be 1.5.
  • ...and 2 more figures