Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, Dong Yu

TL;DR

This work tackles the exploration inefficiency and entropy collapse observed in RLVR for large language models by introducing Curiosity-Driven Exploration (CDE). CDE leverages two intrinsic signals: actor perplexity as a curiosity bonus and a multi-head bootstrap critic’s posterior variance as another exploration guide, integrated into PPO/GRPO frameworks with carefully calibrated clipping. Theoretical results link the actor-based perplexity bonus to penalizing overconfident errors and promoting diverse correct responses, while the critic bonus aligns with count-based exploration in linear MDPs. Empirically, CDE yields notable gains (roughly +3 points on AIME benchmarks) and improves calibration, providing a scalable, minimally invasive enhancement to RLVR training. The findings offer a principled perspective on balancing exploration and exploitation in LLM reasoning and highlight calibration as a key facet of robust RLVR systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.

CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

TL;DR

This work tackles the exploration inefficiency and entropy collapse observed in RLVR for large language models by introducing Curiosity-Driven Exploration (CDE). CDE leverages two intrinsic signals: actor perplexity as a curiosity bonus and a multi-head bootstrap critic’s posterior variance as another exploration guide, integrated into PPO/GRPO frameworks with carefully calibrated clipping. Theoretical results link the actor-based perplexity bonus to penalizing overconfident errors and promoting diverse correct responses, while the critic bonus aligns with count-based exploration in linear MDPs. Empirically, CDE yields notable gains (roughly +3 points on AIME benchmarks) and improves calibration, providing a scalable, minimally invasive enhancement to RLVR training. The findings offer a principled perspective on balancing exploration and exploitation in LLM reasoning and highlight calibration as a key facet of robust RLVR systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.

Paper Structure

This paper contains 24 sections, 4 theorems, 36 equations, 11 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries: RLVR, GRPO and PPO
  3. Group Relative Policy Optimization (GRPO, shao2024deepseekmath)
  4. Proximal Policy Optimization (PPO, schulman2017proximal)
  5. CDE: Curiosity-Driven Exploration
  6. Challenge of Count-based exploration for RLVR
  7. Exploration Guided by Actor Curiosity
  8. Intuitions and Theoretical Foundation
  9. Exploration Guided by Critic Curiosity
  10. Experiments
  11. Dataset and Model
  12. Main Results
  13. Understanding the Effect of the PPL Bonus
  14. Further Analysis of the Multi-Head Critic
  15. Related Work
  16. ...and 9 more sections

Key Result

Theorem 3.1

Let $\pi_t$ denote the policy at training step t. With PPL bonus in Equation eq:actor_bonus, the update to $\pi_{t+1}$ calibrates the policy's confidence as follows: (i) Among correct responses, trajectories with higher perplexity receive a larger relative probability increase. (ii) Among incorrect

Figures (11)

  • Figure 1: Distribution of number of visitations across hash-cells.
  • Figure 2: Responses by correctness and avg PPL.
  • Figure 3: An illustration of the model's policy distribution for selecting the next token.
  • Figure 4: An illustration of two posterior distributions of the critic and their bootstrap approximations.
  • Figure 5: Illustration of the multi-head critic framework.
  • ...and 6 more figures

Theorems & Definitions (5)

  • Theorem 3.1
  • Theorem 3.2
  • Lemma E.2: Proposition 2.3 in jin2020provably
  • Theorem E.3
  • proof