Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

First Return, Entropy-Eliciting Explore

Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, Zejun Ma

TL;DR

FR3E addresses unstable exploration in RL-based reasoning for LLMs by pinpointing high-uncertainty decision points along base trajectories and performing targeted, semantically grounded rollouts. It combines entropy-driven block construction with diversified path sampling and an adaptive advantage modulation to stabilize learning without dense supervision. Empirical results on AIME24 and other benchmarks show improved training stability, longer coherent reasoning, and higher proportions of fully correct trajectories, especially for generalist LLM backbones, with smaller gains on math-specialized models. The framework builds on Go-Explore ideas, adapts PPO clipping to preserve exploration, and demonstrates robust trajectory-level reward shaping through structured exploration.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.

First Return, Entropy-Eliciting Explore

TL;DR

FR3E addresses unstable exploration in RL-based reasoning for LLMs by pinpointing high-uncertainty decision points along base trajectories and performing targeted, semantically grounded rollouts. It combines entropy-driven block construction with diversified path sampling and an adaptive advantage modulation to stabilize learning without dense supervision. Empirical results on AIME24 and other benchmarks show improved training stability, longer coherent reasoning, and higher proportions of fully correct trajectories, especially for generalist LLM backbones, with smaller gains on math-specialized models. The framework builds on Go-Explore ideas, adapts PPO clipping to preserve exploration, and demonstrates robust trajectory-level reward shaping through structured exploration.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.

Paper Structure

This paper contains 23 sections, 17 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. MDP Setup for Autoregressive Generation
  5. Rejection Sampling
  6. Clip-Higher
  7. Structured Exploration
  8. FR3E: First Return, Entropy-Eliciting Explore
  9. First Return: Block Definition via Uncertainty Signals
  10. Base Trajectory Generation
  11. Entropy Computation
  12. Entropy-Sensitive Position Selection
  13. Semantic Block & State Construction
  14. Entropy-Eliciting Explore: Diversified Path Sampling from Anchors
  15. Adaptive Advantage Modulation for Stable Learning
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Overview of the FR3E framework. Stage 1: First Return begins with base trajectory generation from query $q$, followed by token-wise entropy computation to identify high-uncertainty positions. These positions serve as segmentation points for constructing intermediate semantic states $S_j$. Stage 2: Entropy-Eliciting Explore launches multiple rollouts from each state $S_j$, evaluates the reward for each, and computes empirical values $V(S_j)$ to guide adaptive policy updates. This two-stage design encourages diverse yet structured exploration based on model uncertainty signals.
  • Figure 2: Frequent tokens with the highest average entropy
  • Figure 3: Entropy Loss Comparison Across Models
  • Figure 4: AIME24 Comparison Across Different Models
  • Figure 5: Advantage Comparison Across Different Models
  • ...and 4 more figures