ToLeaP: Rethinking Development of Tool Learning with Large Language Models

TL;DR

This work introduces ToLeaP, a standardized, one-click evaluation platform for tool-learning benchmarks in large language models. By reproducing 33 benchmarks across 41 LLMs and compiling 21 training datasets, the authors provide a holistic view of tool-use capabilities and bottlenecks, highlighting four core challenges: benchmark limitations, autonomous data learning, generalization to unseen tools, and long-horizon task solving. The paper also proposes concrete directions to address these challenges, including real-world benchmark construction, compatibility-aware autonomous learning, rationale-based thinking to improve generalization, and explicit key-clue recall for long-horizon planning, with preliminary evidence supporting their potential. The platform and findings offer a unified lens for researchers to measure progress, inform dataset design, and accelerate the development of robust, real-world tool-learning LLMs. This work has practical significance for advancing AI systems that can autonomously interact with external tools in dynamic environments, ultimately boosting reliability and real-world applicability.

Abstract

Tool learning, which enables large language models (LLMs) to utilize external tools effectively, has garnered increasing attention for its potential to revolutionize productivity across industries. Despite rapid development in tool learning, key challenges and opportunities remain understudied, limiting deeper insights and future advancements. In this paper, we investigate the tool learning ability of 41 prevalent LLMs by reproducing 33 benchmarks and enabling one-click evaluation for seven of them, forming a Tool Learning Platform named ToLeaP. We also collect 21 out of 33 potential training datasets to facilitate future exploration. After analyzing over 3,000 bad cases of 41 LLMs based on ToLeaP, we identify four main critical challenges: (1) benchmark limitations induce both the neglect and lack of (2) autonomous learning, (3) generalization, and (4) long-horizon task-solving capabilities of LLMs. To aid future advancements, we take a step further toward exploring potential directions, namely (1) real-world benchmark construction, (2) compatibility-aware autonomous learning, (3) rationale learning by thinking, and (4) identifying and recalling key clues. The preliminary experiments demonstrate their effectiveness, highlighting the need for further research and exploration.

Figures (11)

• Figure 1: A common road map of tool learning research.
• Figure 2: Overall model performance on ToLeaP. Due to the limited space, we calculate the sample-size-weighted average performance across the total 64 metrics from all the benchmarks in ToLeaP.
• Figure 3: Model performance on RotBench and Sealtools. The comparison of same LLM trained on selected data (ToolACE) versus full data SFT and DPO.
• Figure 4: An illustration of shortcut learning. The LLaMa-3.1-8B-instruct-based models forward the information provided in the prompt directly as tool parameters, without performing any modifications and validity checks.
• Figure 5: Model performance on 200 examples from the BFCL-v3 Multi-Turn task. We manually review all bad cases for accuracy.