From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions

TL;DR

The paper addresses the misalignment between LLMs and human-centric tool documentation, which hinders effective tool learning. It introduces DRAFT, a self-driven framework that dynamically refines tool documentation through three phases—experience gathering, learning from experience, and documentation rewriting—augmented by diversity-promoting exploration and a tool-adaptive termination mechanism. Through extensive experiments on ToolBench and RestBench, DRAFT improves documentation quality and enhances cross-model generalization, leading to better tool usage by LLMs and heightened robustness across models. The work demonstrates that automated, feedback-driven documentation refinement can adapt to evolving tool capabilities, offering practical impact for real-world LLM tool usage and deployment.

Abstract

Tool learning enables Large Language Models (LLMs) to interact with external environments by invoking tools, serving as an effective strategy to mitigate the limitations inherent in their pre-training data. In this process, tool documentation plays a crucial role by providing usage instructions for LLMs, thereby facilitating effective tool utilization. This paper concentrates on the critical challenge of bridging the comprehension gap between LLMs and external tools due to the inadequacies and inaccuracies inherent in existing human-centric tool documentation. We propose a novel framework, DRAFT, aimed at Dynamically Refining tool documentation through the Analysis of Feedback and Trials emanating from LLMs' interactions with external tools. This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases: experience gathering, learning from experience, and documentation rewriting, to iteratively enhance the tool documentation. This process is further optimized by implementing a diversity-promoting exploration strategy to ensure explorative diversity and a tool-adaptive termination mechanism to prevent overfitting while enhancing efficiency. Extensive experiments on multiple datasets demonstrate that DRAFT's iterative, feedback-based refinement significantly ameliorates documentation quality, fostering a deeper comprehension and more effective utilization of tools by LLMs. Notably, our analysis reveals that the tool documentation refined via our approach demonstrates robust cross-model generalization capabilities.

Figures (7)

• Figure 1: (a): An illustration showcasing issues in current tool documentation, including incomplete, redundant, and inaccurate information. (b): An illustration of how LLMs incrementally enhance their understanding of tool usage through iterative exploration, transitioning from untested functionalities (gray) to discovered capabilities (white) via a trial-and-error process. (c): The comparison between the raw and improved tool documentation regarding its usefulness on three tool learning datasets, highlighting that DRAFT is able to produce tool documentation that is more favored by LLMs.
• Figure 2: Schematic illustration of our proposed self-driven iterative improvement framework, DRAFT. The left part illustrates the three distinct learning phases in our framework, where a trial refers to a single iteration of the process. The middle part shows the two specialized mechanisms designed to optimize the process. The right part depicts an example of using DRAFT to iteratively modify the tool documentation (a complete revision trajectory is presented in Appendix \ref{['app:revision']}).
• Figure 3: Prompt template for explorer.
• Figure 4: Prompt template for analyzer.
• Figure 5: Prompt template for rewriter.