Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs

TL;DR

The paper argues that low-entropy segments, not just high-entropy tokens, encode stable, reusable reasoning structures in LLMs. It introduces LESS, a correctness-aware, segment-level advantage shaping framework that amplifies low-entropy segments unique to correct trajectories, suppresses those unique to incorrect ones, and neutralizes shared segments, while preserving high-entropy exploration. Instantiated on GRPO, LESS improves accuracy and robustness across six math benchmarks and three backbones, and reduces worst-case dispersion among rollouts. The findings suggest that leveraging stable, low-entropy reasoning patterns offers a principled path to more reliable RLVR for reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a central approach for improving the reasoning ability of large language models. Recent work studies RLVR through token entropy, arguing that high-entropy tokens drive exploration and should receive stronger updates. However, they overlook the fact that most of a reasoning trajectory consists of low-entropy segments that encode stable and reusable structural patterns. Through qualitative and quantitative analyses, we find that the overlap of low-entropy segments across correct responses strongly correlates with model accuracy, while overlaps involving incorrect responses exhibit stable but unproductive patterns. Motivated by these findings, we propose LESS, a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments. LESS amplifies segments unique to correct responses, suppresses those unique to incorrect ones, and neutralizes segments shared by both, while preserving high-entropy exploration in the underlying RL algorithm. Instantiated on top of the popular GRPO, LESS consistently improves accuracy over strong RL baselines across three backbones and six math benchmarks, achieves stronger robustness of the performance floor.

Figures (7)

• Figure 1: Low-entropy analysis reveals stable reasoning behaviors. Left: a case study where correct and incorrect responses exhibit shared and distinct low-entropy segments. Right-top: Across math benchmarks, accuracy strongly correlates with low-entropy segment overlap in correct responses (Pearson r, p). Right-bottom: During GRPO training of Qwen2.5-Math-7B, both accuracy and low-entropy overlap rise together, showing that performance gains emerge alongside the stabilization of reasoning patterns.
• Figure 2: Training dynamics (accuracy over training) of GRPO and LESS across three backbones.
• Figure 3: Training-dynamics comparison between LESS and GRPO across three model sizes. Top: Ratio of low-entropy segments that overlap exclusively among correct responses (higher is better). Bottom: Ratio between the entropy of incorrect responses and correct responses (higher indicates that incorrect answers remain exploratory). LESS consistently strengthens productive low-entropy structures while preventing premature entropy collapse in incorrect trajectories.
• Figure 4: Standard deviation of 32 sampled responses (std@32). LESS reduces response-level variability across all backbones, indicating more stable and less volatile reasoning behavior compared with GRPO.
• Figure 5: Effect of the low-entropy segment-length threshold $\mu$ on training dynamics for Qwen2.5-Math-7B. We compare $\mu=3,5,7$ and report accuracy over training steps.