Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExGRPO: Learning to Reason from Experience

Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, Yu Cheng

TL;DR

ExGRPO tackles the inefficiency of on-policy RLVR by introducing principled experience management that prioritizes high-value reasoning experiences. It buckets past experiences by question difficulty, selects low-entropy trajectories, and blends off-policy replay with on-policy updates via a mixed objective, aided by policy shaping and a delayed start. Empirical results across 1.5B–8B parameter backbones show ExGRPO consistently improves reasoning performance on both in-distribution and out-of-distribution benchmarks and stabilizes training on weaker models. The work demonstrates that careful experience management can meaningfully boost data efficiency and scalability in RLVR for large language models, marking a key step toward practical, scalable reasoning AI.

Abstract

Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.

ExGRPO: Learning to Reason from Experience

TL;DR

ExGRPO tackles the inefficiency of on-policy RLVR by introducing principled experience management that prioritizes high-value reasoning experiences. It buckets past experiences by question difficulty, selects low-entropy trajectories, and blends off-policy replay with on-policy updates via a mixed objective, aided by policy shaping and a delayed start. Empirical results across 1.5B–8B parameter backbones show ExGRPO consistently improves reasoning performance on both in-distribution and out-of-distribution benchmarks and stabilizes training on weaker models. The work demonstrates that careful experience management can meaningfully boost data efficiency and scalability in RLVR for large language models, marking a key step toward practical, scalable reasoning AI.

Abstract

Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.

Paper Structure

This paper contains 75 sections, 2 theorems, 53 equations, 11 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning with Verifiable Rewards.
  4. Experience-based Reinforcement Learning.
  5. Preliminaries
  6. Reinforcement Learning with Verifiable Reward
  7. Verifiable Reward Function.
  8. Group Relative Policy Optimization (GRPO).
  9. Preliminary Study on Experience Data
  10. Setup.
  11. Are all questions equally useful for training?
  12. Does lower entropy imply better reasoning?
  13. Low-entropy trajectories mitigate misguided learning.
  14. Methodology
  15. ExGRPO: Experience Management
  16. ...and 60 more sections

Key Result

Theorem 1

Under Assumption A1, let $\mathcal{G}_{q^*}=\{o^*\}\cup\{o_i\}_{i=1}^{K-1}$ be a mixed group where $o^*$ was sampled from $\pi_{\theta_{\text{past}}}$ and $\{o_i\}$ were sampled from $\pi_{\theta_{\text{old}}}$. For any measurable function $g$ of a trajectory and its group (e.g. $g(o,\mathcal{G})=\s where the right hand side is the expectation with $o^*$ replaced by sampling $\tilde{o}\sim\pi_\the

Figures (11)

  • Figure 1: Analysis of question difficulty and entropy in on-policy RLVR training: (a) Test performance of models trained on different question groups; (b) Entropy distributions of logically correct trajectories across question groups; (c) Entropy comparison between correct and incorrect trajectories.
  • Figure 2: Overview of Experiential Group Relative Policy Optimization (ExGRPO). ExGRPO operates in two phases: (a) Experience Management and (b) Policy Optimization (cf. \ref{['alg:bucket-lowH']}).
  • Figure 3: A comparison of benchmark performance for different backbone models and training variants, showing performance on both in-distribution and out-of-distribution tasks (cf. \ref{['sec:model-ext-nums']}).
  • Figure 4: Learning dynamics of On-Policy vs. ExGRPO during training Llama-3.1 8B. ExGRPO stabilizes training and achieves higher rewards, while on-policy suffers from training collapse.
  • Figure 5: Dynamics of experience replay buffer and retried set.
  • ...and 6 more figures

Theorems & Definitions (4)

  • Theorem 1: Unbiasedness
  • proof : Proof Sketch
  • Proposition 2: Variance upper bound for experiential term
  • proof : Proof Sketch