Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration

Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, Zhiqiang Zhang

TL;DR

This work addresses the data-inefficiency of reinforcement learning with verifiable rewards (RLVR) for large language models by exploiting intrinsic data properties. It introduces PREPO, a method that combines a perplexity-based schedule for selecting prompts with a sequence-level entropy weighting to maintain exploration during rollout generation. Empirical results on Qwen and Llama show PREPO can reduce rollouts by up to 3x while preserving or improving performance on mathematical reasoning benchmarks, supported by ablations and theoretical insights. The approach advances practical data efficiency in RLVR and suggests broader opportunities for intrinsic data signals to guide learning in large-scale language models.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer rollouts than the baselines. Beyond empirical gains, we provide theoretical and in-depth analyses explaining the underlying rationale of our method to improve the data efficiency of RLVR.

Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration

TL;DR

This work addresses the data-inefficiency of reinforcement learning with verifiable rewards (RLVR) for large language models by exploiting intrinsic data properties. It introduces PREPO, a method that combines a perplexity-based schedule for selecting prompts with a sequence-level entropy weighting to maintain exploration during rollout generation. Empirical results on Qwen and Llama show PREPO can reduce rollouts by up to 3x while preserving or improving performance on mathematical reasoning benchmarks, supported by ablations and theoretical insights. The approach advances practical data efficiency in RLVR and suggests broader opportunities for intrinsic data signals to guide learning in large-scale language models.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer rollouts than the baselines. Beyond empirical gains, we provide theoretical and in-depth analyses explaining the underlying rationale of our method to improve the data efficiency of RLVR.

Paper Structure

This paper contains 34 sections, 14 equations, 20 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary Analysis
  3. Prompts with Lower-PPL Tend to Yield Higher Passrate
  4. Training Dynamics of Low-PPL and High-PPL Prompts
  5. PREPO: PPL-Schedule Relative-Entropy Policy Optimization
  6. General Online Batch Selection
  7. PPL-Schedule Online Batch Selection
  8. Relative-Entropy Weighting
  9. Objective Function
  10. Related Work
  11. Data efficiency in RLVR.
  12. Entropy Mechanism in RLVR.
  13. Experiments
  14. Main Results
  15. Ablation Study
  16. ...and 19 more sections

Figures (20)

  • Figure 1: Comparison of PREPO and GRPO with random 20% selection on Qwen2.5-Math-7B, averaged across AIME24, AIME25, MATH-500, and Olympiad Bench.
  • Figure 2: Prompt PPL versus average passrate@16.
  • Figure 3: Training dynamics of Low-PPL vs. High-PPL prompts on Qwen2.5-Math-7B. (a) High-PPL prompts have higher entropy. (b) Low-PPL prompts have more reward gains. (c) Low-PPL prompts reach higher all-correct ratios faster. (d) Low-PPL prompts show higher zero-advantage ratios in the later stage. (e) High-PPL prompts eventually outperform Low-PPL prompts on AIME24 aime2024I.
  • Figure 4: Comparison among Low-PPL, High-PPL, and Random Subsets. Random lies between the two, showing that PPL-based grouping provides a meaningful pruning signal.
  • Figure 5: Comparison of training dynamics between PPL-schedule and static PPL selection (Low- and High-PPL groups)
  • ...and 15 more figures