Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios

TL;DR

UltraTool introduces a comprehensive benchmark for evaluating LLMs on end-to-end tool utilization in real-world, multi-domain tasks. It formalizes samples as (Q, P, T) with hierarchical NL planning and supports tool creation, including the generation of new tools, without reliance on predefined toolsets. The framework uses GPT-4-based NL planning annotation, automatic refinement, and a multi-dimensional LLM-as-Judge scoring method, alongside skeleton tool designs to simulate non-executable tools. Empirical results show GPT-4 achieving the highest performance and reveal clear gaps in open-source models for planning, tool creation, and usage, while confirming alignment with human judgments. The work offers a detailed, transferable methodology for evaluating future LLMs in real-world tool utilization scenarios and sets a baseline for progress in tool-aware AI systems.

Abstract

The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs' ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.

Figures (33)

• Figure 1: Illustration of (a) tool utilization process in real-world complex scenarios and (b) construction process of previous benchmarks.
• Figure 2: Data example of UltraTool, including user query, tree-structure planning, and toolset.
• Figure 3: Specific domain distribution of UltraTool.
• Figure 4: The overall construction process of UltraTool, including (a) query collection, (b) solution annotation, and (c) manual refinement.
• Figure 5: Multi-dimensional scores of planning and tool creation evaluation of 4 representative models under different model scales, including: Qwen-7B, Qwen-14B, Qwen-72B and GPT-4.