R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning

TL;DR

R\u00b3 addresses intra-group advantage collapse in reinforcement learning for LLM reasoning by combining three mechanisms: Cross-Context Replay to preserve batch diversity, In-Context Self-Reflection to refine hard queries with historical failures, and Structural Entropy Ranking Reward to provide informative signals from truncated trajectories. It relies on a centralized sample buffer to fuse on- and off-policy trajectories, enabling stable learning without process rewards and delivering state-of-the-art results on multiple math benchmarks with reduced reasoning tokens. Using a DeepSeek-R1-Distill-Qwen-1.5B base model and the DeepscaleR-40k math dataset, R\u00b3 achieves strong performance (e.g., average scores around 60-67 on AIME/MINERVA-like benchmarks) across AIME24, MATH500, AMC 2023, Minerva Math, and OlympiadBench. The approach demonstrates substantial improvements in sample efficiency and reasoning quality, offering a scalable path toward more reliable mathematical reasoning in LLMs.

Abstract

Large reasoning models (LRMs) aim to solve diverse and complex problems through structured reasoning. Recent advances in group-based policy optimization methods have shown promise in enabling stable advantage estimation without reliance on process-level annotations. However, these methods rely on advantage gaps induced by high-quality samples within the same batch, which makes the training process fragile and inefficient when intra-group advantages collapse under challenging tasks. To address these problems, we propose a reinforcement learning mechanism named \emph{\textbf{R^3}} that along three directions: (1) a \emph{cross-context \underline{\textbf{R}}eplay} strategy that maintains the intra-group advantage by recalling valuable examples from historical trajectories of the same query, (2) an \emph{in-context self-\underline{\textbf{R}}eflection} mechanism enabling models to refine outputs by leveraging past failures, and (3) a \emph{structural entropy \underline{\textbf{R}}anking reward}, which assigns relative rewards to truncated or failed samples by ranking responses based on token-level entropy patterns, capturing both local exploration and global stability. We implement our method on Deepseek-R1-Distill-Qwen-1.5B and train it on the DeepscaleR-40k in the math domain. Experiments demonstrate our method achieves SoTA performance on several math benchmarks, representing significant improvements and fewer reasoning tokens over the base models. Code and model will be released.

Figures (4)

• Figure 1: Multiple responses within the same group may exhibit little to no reward variation, leading to inefficient training or premature stagnation (left). Maintaining meaningful differences among intra-group samples helps preserve advantage gaps and alleviates this problem (right).
• Figure 2: Overview of the R³ architecture. From left to right: (1) For hard queries from prior rounds, perform in-context self-reflection by retrieving historical samples from the sample buffer; (2) The policy model outputs a response, which is then passed through a verifier to assess its quality; (3) Cross-context replay leverages historical samples from the buffer to enhance advantage estimation; (4) Truncated responses are evaluated using the structural entropy ranking reward, which is then used for advantage computation.
• Figure 3: Performance evaluation on challenging subsets of AIME24 and MATH
• Figure 4: Training Dynamics and Entropy Evolution Analysis