Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning

Jingchu Wang, Bingbing Xu, Yige Yuan, Bin Xie, Xiaoqian Sun, Huawei Shen

TL;DR

The paper addresses the conflict between generating stable inference responses and exploring diverse training trajectories in RL for LLM reasoning. It introduces R^2PO, which adds a Residual Rollout-Head on top of a frozen backbone to decouple exploration from exploitation, enabling controlled, diverse trajectory generation while preserving high-quality inference. Through a two-stage iterative training regime and a rule-based Correctness/Formatting reward plus a Group Inverse-Frequency reward, R^2PO achieves consistent gains (e.g., +3.1% on MATH-500 and +2.4% on APPS) and exhibits improved robustness to reward misspecification and reduced length bias across GSM8K, MATH-500, MBPP, HumanEval, and APPS using Qwen backbones (3B and 8B). The approach remains computationally efficient at inference (no Rollout-Head during deployment) and demonstrates scalability and generalization across tasks, offering a practical strategy for stable, high-quality reasoning in large-scale LLM RL settings.

Abstract

Reinforcement learning has become a central paradigm for improving LLM reasoning. However, existing methods use a single policy to produce both inference responses and training optimization trajectories. The objective conflict between generating stable inference responses and diverse training trajectories leads to insufficient exploration, which harms reasoning capability. In this paper, to address the problem, we propose R$^2$PO (Residual Rollout Policy Optimization), which introduces a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, enabling controlled trajectory diversification during training while keeping inference generation stable. Experiments across multiple benchmarks show that our method consistently outperforms baselines, achieving average accuracy gains of 3.1% on MATH-500 and 2.4% on APPS, while also reducing formatting errors and mitigating length bias for stable optimization. Our code is publicly available at https://github.com/RRPO-ARR/Code.

R$^2$PO: Decoupling Training Trajectories from Inference Responses for LLM Reasoning

TL;DR

The paper addresses the conflict between generating stable inference responses and exploring diverse training trajectories in RL for LLM reasoning. It introduces R^2PO, which adds a Residual Rollout-Head on top of a frozen backbone to decouple exploration from exploitation, enabling controlled, diverse trajectory generation while preserving high-quality inference. Through a two-stage iterative training regime and a rule-based Correctness/Formatting reward plus a Group Inverse-Frequency reward, R^2PO achieves consistent gains (e.g., +3.1% on MATH-500 and +2.4% on APPS) and exhibits improved robustness to reward misspecification and reduced length bias across GSM8K, MATH-500, MBPP, HumanEval, and APPS using Qwen backbones (3B and 8B). The approach remains computationally efficient at inference (no Rollout-Head during deployment) and demonstrates scalability and generalization across tasks, offering a practical strategy for stable, high-quality reasoning in large-scale LLM RL settings.

Abstract

Reinforcement learning has become a central paradigm for improving LLM reasoning. However, existing methods use a single policy to produce both inference responses and training optimization trajectories. The objective conflict between generating stable inference responses and diverse training trajectories leads to insufficient exploration, which harms reasoning capability. In this paper, to address the problem, we propose RPO (Residual Rollout Policy Optimization), which introduces a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, enabling controlled trajectory diversification during training while keeping inference generation stable. Experiments across multiple benchmarks show that our method consistently outperforms baselines, achieving average accuracy gains of 3.1% on MATH-500 and 2.4% on APPS, while also reducing formatting errors and mitigating length bias for stable optimization. Our code is publicly available at https://github.com/RRPO-ARR/Code.
Paper Structure (39 sections, 8 equations, 3 figures, 8 tables)

This paper contains 39 sections, 8 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. RL for Reasoning LLMs
  4. Exploration-Exploitation Dilemma
  5. Decoupled and Auxiliary Architectures
  6. Method
  7. Motivation
  8. Overall Architecture
  9. Reward Design
  10. Correctness and Formatting Reward.
  11. Group Inverse-Frequency Reward.
  12. Iterative Training
  13. Rollout-Head Optimization
  14. Main Policy Optimization
  15. Discussion
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Decoupling Ideal Responses and Optimization Trajectories. Left: The mismatch between ideal inference responses and ideal training trajectories. Right: Single-head policy optimization versus R$^2$PO with a decoupled Rollout-Head for stable exploration.
  • Figure 2: Overview of R$^2$PO. An iterative loop of two alternating stages: Stage 1 (top) optimizes the Rollout-Head using GIF reward with a frozen backbone; Stage 2 (bottom) updates the base policy on task rewards using the Rollout-Head as a fixed behavioral sampler.
  • Figure 3: The mean and variance of rewards during the training process on GSM8K. The upper row shows the mean reward, while the lower row displays the variance of the mean reward. “Info” refers to Informative Rollouts, which contain varying rewards and thus supply non-zero advantage signals for policy updates.