Tools are under-documented: Simple Document Expansion Boosts Tool Retrieval

TL;DR

Tool-DE targets the persistent semantic gap in tool retrieval caused by under-documented tools by introducing a scalable document expansion pipeline that enriches tool profiles with structured fields. It couples this with two specialized models, Tool-Embed (dense retriever) and Tool-Rank (reranker), trained on large expanded corpora to achieve state-of-the-art results on Tool-DE and ToolRet benchmarks. The work demonstrates that expanding documentation improves both retrieval and reranking, while also revealing when extra fields help or hurt performance and how expansion affects training signals and evaluation. Overall, Tool-DE provides a principled foundation for future research in scalable, accurate tool retrieval and tool-augmented intelligence.

Abstract

Large Language Models (LLMs) have recently demonstrated strong capabilities in tool use, yet progress in tool retrieval remains hindered by incomplete and heterogeneous tool documentation. To address this challenge, we introduce Tool-DE, a new benchmark and framework that systematically enriches tool documentation with structured fields to enable more effective tool retrieval, together with two dedicated models, Tool-Embed and Tool-Rank. We design a scalable document expansion pipeline that leverages both open- and closed-source LLMs to generate, validate, and refine enriched tool profiles at low cost, producing large-scale corpora with 50k instances for embedding-based retrievers and 200k for rerankers. On top of this data, we develop two models specifically tailored for tool retrieval: Tool-Embed, a dense retriever, and Tool-Rank, an LLM-based reranker. Extensive experiments on ToolRet and Tool-DE demonstrate that document expansion substantially improves retrieval performance, with Tool-Embed and Tool-Rank achieving new state-of-the-art results on both benchmarks. We further analyze the contribution of individual fields to retrieval effectiveness, as well as the broader impact of document expansion on both training and evaluation. Overall, our findings highlight both the promise and limitations of LLM-driven document expansion, positioning Tool-DE, along with the proposed Tool-Embed and Tool-Rank, as a foundation for future research in tool retrieval.

Figures (11)

• Figure 1: Document expansion pipeline for constructing Tool-DE.
• Figure 2: Impact of individual fields on retrieval: example-usage brings minimal gains in gritlm and bm25s, and even hurts performance in gritlm, while function and when-to-use contribute relatively larger improvements.
• Figure 3: Mean similarity scores of positive and negative in Tool-DE and ToolRet.
• Figure 4: Field coverage across the 35 datasets in the ToolRet benchmark.
• Figure 5: Prompt template used for LLM-based completeness auditing of tool documents.