Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning

Siyu Gong, Linan Yue, Weibo Gao, Fangzhou Yao, Shimin Di, Lei Feng, Min-Ling Zhang

TL;DR

AutoTraj tackles the limitations of existing two-stage TIR by automatically repairing low-quality synthesized tool-use trajectories and introducing trajectory-level rewards for RL. It constructs a high-quality SFT dataset through synthesis, evaluation, and repair, and a self-supervised trajectory-pair dataset to train a trajectory reward model. The RL stage then combines format, outcome, and trajectory rewards with GRPO to guide learning toward reliable, concise reasoning paths. Across mathematical and knowledge-intensive benchmarks, AutoTraj achieves superior performance and significantly improved reasoning efficiency, demonstrating scalable, robust tool-integrated reasoning.

Abstract

Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to solve complex tasks by interacting with external tools, yet existing approaches depend on high-quality synthesized trajectories selected by scoring functions and sparse outcome-based rewards, providing limited and biased supervision for learning TIR. To address these challenges, in this paper, we propose AutoTraj, a two-stage framework that automatically learns TIR by repairing and rewarding tool-use trajectories. Specifically, in the supervised fine-tuning (SFT) stage, AutoTraj generates multiple candidate tool-use trajectories for each query and evaluates them along multiple dimensions. High-quality trajectories are directly retained, while low-quality ones are repaired using a LLM (i.e., LLM-as-Repairer). The resulting repaired and high-quality trajectories form a synthetic SFT dataset, while each repaired trajectory paired with its original low-quality counterpart constitutes a dataset for trajectory preference modeling. In the reinforcement learning (RL) stage, based on the preference dataset, we train a trajectory-level reward model to assess the quality of reasoning paths and combine it with outcome and format rewards, thereby explicitly guiding the optimization toward reliable TIR behaviors. Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj in TIR.

Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning

TL;DR

AutoTraj tackles the limitations of existing two-stage TIR by automatically repairing low-quality synthesized tool-use trajectories and introducing trajectory-level rewards for RL. It constructs a high-quality SFT dataset through synthesis, evaluation, and repair, and a self-supervised trajectory-pair dataset to train a trajectory reward model. The RL stage then combines format, outcome, and trajectory rewards with GRPO to guide learning toward reliable, concise reasoning paths. Across mathematical and knowledge-intensive benchmarks, AutoTraj achieves superior performance and significantly improved reasoning efficiency, demonstrating scalable, robust tool-integrated reasoning.

Abstract

Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to solve complex tasks by interacting with external tools, yet existing approaches depend on high-quality synthesized trajectories selected by scoring functions and sparse outcome-based rewards, providing limited and biased supervision for learning TIR. To address these challenges, in this paper, we propose AutoTraj, a two-stage framework that automatically learns TIR by repairing and rewarding tool-use trajectories. Specifically, in the supervised fine-tuning (SFT) stage, AutoTraj generates multiple candidate tool-use trajectories for each query and evaluates them along multiple dimensions. High-quality trajectories are directly retained, while low-quality ones are repaired using a LLM (i.e., LLM-as-Repairer). The resulting repaired and high-quality trajectories form a synthetic SFT dataset, while each repaired trajectory paired with its original low-quality counterpart constitutes a dataset for trajectory preference modeling. In the reinforcement learning (RL) stage, based on the preference dataset, we train a trajectory-level reward model to assess the quality of reasoning paths and combine it with outcome and format rewards, thereby explicitly guiding the optimization toward reliable TIR behaviors. Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj in TIR.
Paper Structure (39 sections, 13 equations, 15 figures, 7 tables)

This paper contains 39 sections, 13 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool-Integrated Reasoning
  4. Reinforcement Learning with Verifiable Rewards
  5. Problem Formulation
  6. Framework of AutoTraj
  7. Supervised Fine-Tuning via Synthesized Trajectory
  8. Candidate TIR Trajectory Synthesis
  9. Multi-Dimensional Trajectory Evaluation
  10. LLM-based Trajectory Repair
  11. Cold-Start Supervised Fine-tuning
  12. Trajectory-Aware Reinforcement Learning
  13. Trajectory Reward Model
  14. Reward Design
  15. RL Training with GRPO
  16. ...and 24 more sections

Figures (15)

  • Figure 1: Architecture of AutoTraj. (a) SFT Stage: AutoTraj first synthesizes, evaluates, and repairs tool-use trajectories to construct a high-quality SFT dataset and positive–negative trajectory pairs. (b) RL Stage: A trajectory reward model is then trained on these pairs and integrated with format and outcome rewards under GRPO, providing trajectory-level supervision.
  • Figure 2: Average length of TIR trajectories across benchmarks. Lower values indicate higher reasoning efficiency.
  • Figure 3: Average performance of AutoTraj under different RL training data scales.
  • Figure 4: Question Length Distribution for High-Quality Trajectory Set $\mathcal{T}_\text{high}$ and High-Quality Trajectory Set $\mathcal{T}_\text{high} \cup \tau^{+}$.
  • Figure 5: Validation score of AutoTraj and its ablated variant without trajectory rewards (i.e., AutoTraj w/o TR) during GRPO training.
  • ...and 10 more figures