Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction

Xingjie Gao, Pengcheng Huang, Zhenghao Liu, Yukun Yan, Shuo Wang, Zulong Chen, Chen Qian, Ge Yu, Yu Gu

TL;DR

This paper proposes ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment and significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools.

Abstract

Equipping Large Language Models (LLMs) with external tools enables them to solve complex real-world problems. However, the robustness of existing methods remains a critical challenge when confronting novel or evolving tools. Existing trajectory-centric paradigms primarily rely on memorizing static solution paths during training, which limits the ability of LLMs to generalize tool usage to newly introduced or previously unseen tools. In this paper, we propose ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment. To optimize LLMs for tool planning and invocation, ToolMaster adopts a trial-and-execution paradigm, which trains LLMs to first imitate teacher-generated trajectories containing explicit tool trials and self-correction, followed by reinforcement learning to coordinate the trial and execution phases jointly. This process enables agents to autonomously explore correct tool usage by actively interacting with environments and forming experiential knowledge that benefits tool execution. Experimental results demonstrate that ToolMaster significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools. All code and data are available at https://github.com/NEUIR/ToolMaster.

Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction

TL;DR

This paper proposes ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment and significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools.

Abstract

Equipping Large Language Models (LLMs) with external tools enables them to solve complex real-world problems. However, the robustness of existing methods remains a critical challenge when confronting novel or evolving tools. Existing trajectory-centric paradigms primarily rely on memorizing static solution paths during training, which limits the ability of LLMs to generalize tool usage to newly introduced or previously unseen tools. In this paper, we propose ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment. To optimize LLMs for tool planning and invocation, ToolMaster adopts a trial-and-execution paradigm, which trains LLMs to first imitate teacher-generated trajectories containing explicit tool trials and self-correction, followed by reinforcement learning to coordinate the trial and execution phases jointly. This process enables agents to autonomously explore correct tool usage by actively interacting with environments and forming experiential knowledge that benefits tool execution. Experimental results demonstrate that ToolMaster significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools. All code and data are available at https://github.com/NEUIR/ToolMaster.
Paper Structure (21 sections, 13 equations, 4 figures, 18 tables)

This paper contains 21 sections, 13 equations, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Optimizing LLM Tool Exploration Capability via Environment Interaction
  5. Prompting LLMs to Synthesize Tool Trial Trajectories for SFT
  6. Experimental Methodology
  7. Evaluation Results
  8. Overall Performance
  9. Ablation Study
  10. Effectiveness of ToolMaster in Out-of-Domain Tool-Calling Scenarios
  11. Tool Use Generalization of ToolMaster
  12. Conclusion
  13. Appendix
  14. License
  15. Details of Implementation
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Illustration of the Trial-and-Execution paradigm proposed by ToolMaster.
  • Figure 2: The architecture of ToolMaster.
  • Figure 3: Tool-calling analyses of ToolMaster in out-of-domain scenarios. We use Qwen2.5-7B-Instruct as the backbone model in experiments and conduct experiments on the ToolHop dataset.
  • Figure 4: Characteristics of ToolMaster in tool usage under varying degrees of similarity to the training data. This experiment uses Qwen2.5-7B-Instruct as the backbone model and is evaluated on the ToolHop dataset. Figures \ref{['fig:similarity-sub1']} and \ref{['fig:similarity-sub2']} illustrate the distribution of tools and their calling success rates based on similarity between tool documentation and tools observed during training. Figures \ref{['fig:similarity-sub3']} and \ref{['fig:similarity-sub4']} are plotted over test instances, categorized by the tool with the lowest similarity score in the gold tool set.