Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning

TL;DR

This work addresses the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR) for LLM reasoning. It introduces Exploratory Annealed Decoding (EAD), a simple, plug-and-play intra-sequence temperature schedule that front-loads exploration and gradually exploits to preserve sample quality and training stability. By combining EAD with truncated importance sampling, the approach improves sample efficiency, mitigates entropy collapse, and generalizes across multiple models and RLVR algorithms, with both training-time gains and inference-time benefits. The findings suggest that aligning exploration with the natural dynamics of sequence generation yields robust improvements in reasoning tasks and scalable RLVR deployments. The method is lightweight, broadly compatible, and offers practical benefits for both training efficiency and inference quality.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs), yet its success hinges on effective exploration. An ideal exploration strategy must navigate two fundamental challenges: it must preserve sample quality while also ensuring training stability. While standard fixed-temperature sampling is simple, it struggles to balance these competing demands, as high temperatures degrade sample quality and low temperatures limit discovery. In this work, we propose a simpler and more effective strategy, Exploratory Annealed Decoding (EAD), grounded in the insight that exploration is most impactful on early tokens which define a sequence's semantic direction. EAD implements an intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by annealing the sampling temperature from high to low during generation. This dynamic schedule encourages meaningful, high-level diversity at the start, then gradually lowers the temperature to preserve sample quality and keep the sampling distribution close to the target policy, which is essential for stable training. We demonstrate that EAD is a lightweight, plug-and-play method that significantly improves sample efficiency, consistently outperforming fixed-temperature sampling across various RLVR algorithms and model sizes. Our work suggests that aligning exploration with the natural dynamics of sequential generation offers a robust path to improving LLM reasoning.

Figures (12)

• Figure 1: Average entropy shrinks with output positions for Llama-3-8B-Instruct on MMLU dataset.
• Figure 2: A "forking" experiment on DeepSeek-Llama-3 shows early branching (high-entropy region) yields higher Maj@$k$ on MMLU than late branching (low-entropy region).
• Figure 3: The annealing schedule with different decay rates $d$. A larger $d$ slows the cooling, front-loading exploration over more tokens. We set $c=10, \tau_{\mathrm{max}}=1.2, \tau_{\mathrm{min}}=0.1$ for illustration.
• Figure 4: Pass@16 and Worst@16 performance evaluation in RL training. While EAD improves exploration of high-quality samples (even the worst outperform temperature sampling), the gain diminishes over time; importance sampling can supplement to correct bias and sustain training.
• Figure 5: Pass@16 performance on Qwen-2.5-Math-7B. EAD enables better exploration than fixed-temperature sampling, yielding sustained gains in Pass@16 throughout training.

Theorems & Definitions (4)

• Proposition B.1
• proof
• Proposition C.1
• proof