Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoboGPT-R1: Enhancing Robot Planning with Reinforcement Learning

Jinrui Liu, Bingyan Nie, Boyu Li, Yaran Chen, Yuze Wang, Shunsen He, Haoran Li

TL;DR

The paper addresses the limitations of SFT-based embodied planning in real-world, long-horizon tasks by proposing RoboGPT-R1, a two-stage framework that combines supervised fine-tuning with GRPO-based reinforcement learning. A novel rule-based reward design integrates a structured format component with a dense LCS-based sequence accuracy term to guide long-horizon planning and action ordering. Experiments on EmbodiedBench show substantial gains for a 3B model, outperforming several open- and closed-source baselines and demonstrating strong generalization to unseen environments. The work highlights the effectiveness of dense, verifiable rewards and near-domain data in enabling small models to achieve competitive embodied planning performance and improved long-horizon reasoning.

Abstract

Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully. Despite the success of large language models and vision language models based on Supervised Fine-Tuning (SFT) in planning tasks, they continue facing challenges in performing long-horizon manipulation tasks in complex real-world environments, owing to their restricted common sense and reasoning capabilities. Considering that aligning general-purpose vision language models to robotic planning tasks via supervised fine-tuning suffers from poor generalization and insufficient physical understanding, we propose RoboGPT-R1, a two-stage fine-tuning framework for embodied planning. In this framework, supervised training acquires foundational knowledge through expert sequences, followed by RL to address the model's shortcomings in visual-spatial understanding and reasoning. To achieve physical understanding and action sequence consistency in multi-step reasoning tasks, we design a rule-based reward function that simultaneously considers long-horizon performance and action constraint in the environment. The reasoning model, trained on Qwen2.5-VL-3B, significantly outperforms the larger-scale model, GPT-4o-mini, by 21.33% and surpasses other work trained on Qwen2.5-VL-7B by 20.33% on the EmbodiedBench benchmark.

RoboGPT-R1: Enhancing Robot Planning with Reinforcement Learning

TL;DR

The paper addresses the limitations of SFT-based embodied planning in real-world, long-horizon tasks by proposing RoboGPT-R1, a two-stage framework that combines supervised fine-tuning with GRPO-based reinforcement learning. A novel rule-based reward design integrates a structured format component with a dense LCS-based sequence accuracy term to guide long-horizon planning and action ordering. Experiments on EmbodiedBench show substantial gains for a 3B model, outperforming several open- and closed-source baselines and demonstrating strong generalization to unseen environments. The work highlights the effectiveness of dense, verifiable rewards and near-domain data in enabling small models to achieve competitive embodied planning performance and improved long-horizon reasoning.

Abstract

Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully. Despite the success of large language models and vision language models based on Supervised Fine-Tuning (SFT) in planning tasks, they continue facing challenges in performing long-horizon manipulation tasks in complex real-world environments, owing to their restricted common sense and reasoning capabilities. Considering that aligning general-purpose vision language models to robotic planning tasks via supervised fine-tuning suffers from poor generalization and insufficient physical understanding, we propose RoboGPT-R1, a two-stage fine-tuning framework for embodied planning. In this framework, supervised training acquires foundational knowledge through expert sequences, followed by RL to address the model's shortcomings in visual-spatial understanding and reasoning. To achieve physical understanding and action sequence consistency in multi-step reasoning tasks, we design a rule-based reward function that simultaneously considers long-horizon performance and action constraint in the environment. The reasoning model, trained on Qwen2.5-VL-3B, significantly outperforms the larger-scale model, GPT-4o-mini, by 21.33% and surpasses other work trained on Qwen2.5-VL-7B by 20.33% on the EmbodiedBench benchmark.
Paper Structure (25 sections, 10 equations, 3 figures, 7 tables)

This paper contains 25 sections, 10 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Embodied Planning
  4. Reinforcement Learning for LLMs and VLMs
  5. Methodology
  6. Overview
  7. Data Preparation
  8. Two-stage Training Scheme
  9. Reward Design
  10. Format Reward
  11. LCS Reward
  12. ovrall reward
  13. EXPERIMENTS
  14. Experimental Settings
  15. Main Results
  16. ...and 10 more sections

Figures (3)

  • Figure 1: An overview of RoboGPT-R1. RoboGPT-R1 adopts a two-stage learning paradigm. In the initial phase, supervised fine-tuning introduces the model to data from mathematics, embodied tasks, and visual understanding, establishing a foundation for embodied reasoning. In the second phase, we apply GRPO-based reinforcement fine-tuning guided by a tailored reward function. The model is subsequently evaluated across six categories of tasks, including long-horizon planning and spatial reasoning.
  • Figure 2: Reward Curves in RFT. Our LCS-based accuracy reward provides an appropriate and dense learning signal for RFT, rising steadily from 0.30 to 0.80. The format reward, already aligned by SFT, starts around 0.95 at the onset of RFT and saturates within $\sim$20 steps, stabilizing near 1.0. The overall reward is computed as a weighted sum of the accuracy and format rewards with weights 0.8 and 0.2, respectively, and it increases in tandem with the steady improvement of the accuracy reward.
  • Figure 3: Success rates with different stages. Bars show the macro average and the long-horizon score for the base model, SFT-only, and SFT+RFT model. SFT establishes initial embodied planning (Avg: 1.33%→42.00%) but leaves long-horizon performance limited (26%). Adding RFT lifts the averages further (to 55.33%) and markedly improves long-horizon planning (to 50%), validating the effectiveness of our two-stage framework—SFT for foundational competence and RFT for the additional gains needed to solve long-horizon tasks.