Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees

TL;DR

This paper introduces DART, a reinforcement learning framework that integrates tool-use into long chain-of-thought reasoning without annotated data. It uses a dynamic rollout tree to discover tool-integrated trajectories and a tree-based advantage to credit beneficial sub-trajectories, training with an on-policy objective that preserves the model’s native long-CoT abilities. Empirical results on AIME and GPQA benchmarks show that DART outperforms existing methods, demonstrating improved reasoning quality when tool-use is harmonized with long CoT. The approach offers a scalable, data-efficient path to endow large reasoning models with robust tool-use capabilities while maintaining interpretability and reasoning depth.

Abstract

Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.

Figures (3)

• Figure 1: Overview of our approach on a running example with $M=1$, $N=3$. $\underline{\mathbf{\mathbf{\mathcal{T}_0\rightarrow\mathcal{T}_1\rightarrow...\rightarrow\mathcal{T}_N}}}$: At each expansion step, a promising $(t^*, \bm{h}^*)$ is selected, where $t^*$ is an intermediate position within the existing trajectories (e.g., $\bm{y}$ in $\mathcal{T}_0$), and $\bm{h}^*$ is the corresponding hint. Starting from the prefix $[\bm{y}_{<t*,},\bm{h}^*]$, the policy model $\pi_\theta$ generates a Python snippet $\bm{c}$; the Python interpreter then returns the execution feedback $\bm{o_c}$. $\pi_\theta$ subsequently completes the partial trajectory $[\bm{y}_{<t*,}, \bm{h}^*, \bm{c}, \bm{o}_c]$ by generating $\bm{y}^{'}_{\geq t^*+\mid \bm{h}^*, \bm{c}, \bm{o_c} \mid}$. The new sub-trajectory $\bm{y}^{'}_{\geq t^*}$ is plugged into the tree, while the original $\bm{y}$ is partitioned into $\bm{y}_{<t*}$ and $\bm{y}_{\geq t*}$, finally resulting in $\mathcal{T}_1$. In the illustrated tool‑use example, the code segment employing the sympy package is used to solve a function, replacing the original verbose natural language derivation. Upper right: Each leaf node, together with the nodes along the path from the root to that leaf, constitutes a complete reasoning trajectory. Tokens within a sub‑trajectory inherit the advantage value associated with that sub‑trajectory.
• Figure 2: Metrics of tool-use during training.
• Figure 3: Effect of advantage assignment on the model's tool-use tendency.