TreeRPO: Tree Relative Policy Optimization

TL;DR

TreeRPO presents a reward-model-free framework that uses tree sampling to estimate dense, step-level rewards for LLM reasoning, improving guidance over trajectory-only signals. By propagating leaf rewards upward and forming step-level groups, it delivers fine-grained feedback without separate reward models, building on GRPO’s foundation. Empirical results show substantial Pass@1 gains (e.g., from 19.0% to 35.5%) and reduced token usage (≈18%) on math benchmarks, underscoring both effectiveness and efficiency. These advances offer a scalable approach to enhancing complex reasoning in LLMs through dense, verifiable rewards.

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities through Reinforcement Learning with Verifiable Rewards (RLVR) methods. However, a key limitation of existing approaches is that rewards defined at the full trajectory level provide insufficient guidance for optimizing the intermediate steps of a reasoning process. To address this, we introduce \textbf{\name}, a novel method that estimates the mathematical expectations of rewards at various reasoning steps using tree sampling. Unlike prior methods that rely on a separate step reward model, \name directly estimates these rewards through this sampling process. Building on the group-relative reward training mechanism of GRPO, \name innovatively computes rewards based on step-level groups generated during tree sampling. This advancement allows \name to produce fine-grained and dense reward signals, significantly enhancing the learning process and overall performance of LLMs. Experimental results demonstrate that our \name algorithm substantially improves the average Pass@1 accuracy of Qwen-2.5-Math on test benchmarks, increasing it from 19.0\% to 35.5\%. Furthermore, \name significantly outperforms GRPO by 2.9\% in performance while simultaneously reducing the average response length by 18.1\%, showcasing its effectiveness and efficiency. Our code will be available at \href{https://github.com/yangzhch6/TreeRPO}{https://github.com/yangzhch6/TreeRPO}.

Figures (6)

• Figure 2: The sampling process of our TreeRPO. TreeRPO starts from the question, sampling $N$ nodes at each step until generation is completed or the maximum depth limit $D$ is reached. Then, a verifiable reward function is used to evaluate all leaf nodes and then back-propagates the rewards to their parent nodes, thereby obtaining intermediate step rewards, which achieves process reward signaling. We traverse each node and aggregate all children steps of a node into a group to compute advantages, which are finally formatted into the training batch.
• Figure 3: Performance comparison of our TreeRPO and GRPO on the four selected benchmarks: Math-500, MinervaMath, OlympiadBench, and AIME. The experiments are conducted with Qwen2.5-Math-1.5b, an LLM pretrained with a large amount of mathematical corpus.
• Figure 4: Performance comparison of our TreeRPO and GRPO on the four selected benchmarks: Math-500, MinervaMath, OlympiadBench, and AIME. The experiments are conducted with Qwen2.5-Math-1.5b.
• Figure 5: Response Length comparison of our TreeRPO and GRPO on the four selected benchmarks: Math-500, MinervaMath, OlympiadBench, and AIME. The experiments is conducted with Qwen2.5-Math-1.5b
• Figure 6: Comparison of TreeRPO and GRPO with $bsz = 16$. The pass@1 accuracy and the response length are calculated by taking the average on the four tested benchmarks.