Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models

TL;DR

RL post-training drives final-layer entropy collapse, weakening exploration even when higher-temperature sampling is used. The authors diagnose latent entropy reservoirs in intermediate layers and introduce Latent Exploration Decoding (LED), a training-free decoding scheme that aggregates latent posteriors from the last $d$ layers via top-$k$ filtering and depth-wise cumulative aggregation, selecting the exploration depth by maximal entropy during the DeepThink phase. LED yields consistent improvements in pass@1 and pass@16 across multiple benchmarks and models with negligible overhead, effectively reactivating exploration at higher temperatures. By revealing latent uncertainty in intermediate representations and providing a practical decoding strategy, this work offers a scalable way to recover exploration without retraining large reasoning models.

Abstract

Large Reasoning Models (LRMs) have recently achieved strong mathematical and code reasoning performance through Reinforcement Learning (RL) post-training. However, we show that modern reasoning post-training induces an unintended exploration collapse: temperature-based sampling no longer increases pass@$n$ accuracy. Empirically, the final-layer posterior of post-trained LRMs exhibit sharply reduced entropy, while the entropy of intermediate layers remains relatively high. Motivated by this entropy asymmetry, we propose Latent Exploration Decoding (LED), a depth-conditioned decoding strategy. LED aggregates intermediate posteriors via cumulative sum and selects depth configurations with maximal entropy as exploration candidates. Without additional training or parameters, LED consistently improves pass@1 and pass@16 accuracy by 0.61 and 1.03 percentage points across multiple reasoning benchmarks and models. Project page: https://GitHub.com/Xiaomi-Research/LED.

Figures (8)

• Figure 1: Pass@$n$ accuracy (%) for LLMs under different sampling temperatures, with darker bars representing higher values, specifically, 0.1, 0.3, and 0.6 (temperatures higher than 0.6 are not reported, as they could lead to endless looping and deteriorated performance). For earlier models or non-reasoning models, e.g., QwQ-32B, DeepSeek-8B, and Qwen3-4B-I (Instruct), higher temperature yields higher accuracy, producing a higher accuracy-temperature slope ($\alpha$) as noted in each subtitle. In contrast, for the latest LRMs, MiMo and Qwen3-T (Thinking) series, increasing the temperature could result in negative $\alpha$.
• Figure 2: Normalized entropy across LLM layers.
• Figure 3: The overview of our proposed Latent Exploration Decoding (LED) method.
• Figure 4: Top-$k$ coverage ratios $\{r_k^l\}_{l=1}^L$ for Qwen3-4B-Thinking ($k \in \{1,2,4,8,16\}$, averaged over all benchmarks). Darker colors correspond to greater $k$ values.
• Figure 5: Pass@$n$ accuracy (%) for the latest LRMs with LED under varying sampling temperatures.