Reinforced Efficient Reasoning via Semantically Diverse Exploration

TL;DR

ROSE tackles the limitations of uniform credit assignment and narrow exploration in RLVR for LLMs by introducing a semantic-entropy driven branching mechanism and an ε-exploration strategy within a Monte Carlo Tree Search framework. It also stabilizes learning with a length-aware, segment-level advantage estimator and a KL-regularized training objective to promote concise, correct reasoning. Empirical results on diverse mathematical benchmarks show ROSE delivering substantial performance gains across model scales, while also reducing reasoning length and improving efficiency. The work demonstrates that semantically diversified exploration, combined with length-aware credit, yields meaningful improvements in both effectiveness and efficiency for complex reasoning tasks with LLMs.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has proven effective in enhancing the reasoning of large language models (LLMs). Monte Carlo Tree Search (MCTS)-based extensions improve upon vanilla RLVR (e.g., GRPO) by providing tree-based reasoning rollouts that enable fine-grained and segment-level credit assignment. However, existing methods still suffer from limited exploration diversity and inefficient reasoning. To address the above challenges, we propose reinforced efficient reasoning via semantically diverse explorations, i.e., ROSE, for LLMs. To encourage more diverse reasoning exploration, our method incorporates a semantic-entropy-based branching strategy and an $\varepsilon$-exploration mechanism. The former operates on already sampled reasoning rollouts to capture semantic uncertainty and select branching points with high semantic divergence to generate new successive reasoning paths, whereas the latter stochastically initiates reasoning rollouts from the root, preventing the search process from becoming overly local. To improve efficiency, we design a length-aware segment-level advantage estimator that rewards concise and correct reasoning while penalizing unnecessarily long reasoning chains. Extensive experiments on various mathematical reasoning benchmarks with Qwen and Llama models validate the effectiveness and efficiency of ROSE. Codes are available at https://github.com/ZiqiZhao1/ROSE-rl.

Figures (7)

• Figure 1: (a) Comparison between independent rollout (vanilla GRPO) and MCTS-based rollout. (b) Case study of generation-entropy-based branching. The tokens highlighted in yellow indicate the different tokens generated at the positions of highest entropy. Identical text across different responses is marked with the same colour (green or blue).
• Figure 2: The overview of the ROSE framework. The figure on the left illustrates the structure of the tree-based rollout. Pivot nodes refer to nodes with the highest semantic uncertainty, which are selected according to the semantic entropy. The rollout procedure is detailed in Section \ref{['exploration']}. The figure on the right depicts the advantage estimation pipeline, comprising three stages: (1) node value assignment, (2) segment advantage estimation and (3) length-aware calibration. These stages are described in detail in Section \ref{['advantage']}.
• Figure 3: Learning curves. Average performance across four datasets as training progresses.
• Figure 4: Kernel density estimation (KDE) of pairwise sentence similarities. The dashed line indicates the average cosine similarity.
• Figure 5: The average response length per prompt during the rollout stage (left) and the evaluation stage (right)