The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling

TL;DR

The paper tackles the problem of limited exploration and entropy collapse in RL-trained LLMs by introducing SESA, a sequential-sampling framework. It uses a two-stage training strategy—Stage I for drafting diverse method sketches and Stage II for guided solution generation—to promote diversity and prevent policy collapse, with a GRPO-style update to reinforce novel candidates. Across synthetic, agentic, and reasoning tasks, SESA improves exploration breadth and performance, reviving stalled policies and delivering substantial gains over parallel-sampling baselines. The work offers a practical pathway to more robust and diverse reasoning in RL-trained LLMs, with broad implications for real-world problem solving and AI alignment.

Abstract

Reinforcement learning (RL) has been pivotal in enhancing the reasoning capabilities of large language models (LLMs), but it often suffers from limited exploration and entropy collapse, where models exploit a narrow set of solutions, leading to a loss of sampling diversity and subsequently preventing RL from further improving performance. This issue is exacerbated in parallel sampling methods, where multiple outputs are drawn from the same distribution, potentially causing the model to converge to similar solutions. We propose SESA, a novel SEquential SAmpling framework that mitigates this challenge by generating diverse solution sketches sequentially before expanding them into full reasoning paths. This approach ensures broader exploration by conditioning each new output on previous ones, promoting diversity throughout the process and preventing policy collapse. Our experiments on a synthetic task show that sequential sampling consistently outperforms traditional RL methods in terms of path diversity and recovery from collapse. Further evaluations on real-world tasks demonstrate that SESA improves both the exploration of valid strategies and the overall performance of LLMs. On three agent benchmarks, SESA lifts success rates by $+0.25$, $+0.42$, and $+0.07$ absolute over the base model (up to an additional $211\%$ relative improvement over baseline RL), underscoring its exploration advantage. This work introduces a structured approach to exploration, paving the way for more effective and diverse reasoning in RL-trained LLMs. Our code is released at https://github.com/MuLabPKU/sesa.

Figures (6)

• Figure 1: Parallel sampling quickly plateaus at 11 species, while sequential sampling continues to discover new species, reaching 22 and 80, respectively.
• Figure 2: Training overview. Baseline parallel rollout (left) samples all solutions i.i.d. from the same distribution, while our sequential rollout (right) first generates diverse methods sequentially, then expands each into full solutions in parallel.
• Figure 3: Synthetic path exploration result.
• Figure 4: Performance comparison on Sudoku and AIME24. The results show the effectiveness of our two-stage sequential sampling method in improving solution diversity, particularly in higher Pass@k metrics.
• Figure 5: Comparison of Pass@16/Pass@1 ratios for sequential and parallel sampling. Sequential sampling maintains higher diversity, while parallel sampling declines, indicating a collapse into a "dead policy" with reduced exploration.