Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACEBench: Who Wins the Match Point in Tool Usage?

Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, Wu Liu

TL;DR

ACEBench tackles the lack of realistic, multi-turn tool-usage evaluation for LLMs by introducing Normal, Special, and Agent data categories and an automated sandboxed evaluation system. It covers 2,000 bilingual data entries across 8 domains and 68 sub-domains with 4,538 APIs, enabling end-to-end and granular diagnostics of tool invocation. Experimental results show closed-source models still lead, open-source models are catching up, and fine-tuning may harm generalization to imperfect instructions; Agent tasks remain the hardest benchmark. The framework reduces evaluation cost while providing deeper insights into tool-use capabilities and prompting strategies, with clear pathways for future expansion of scenarios and realism.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.

ACEBench: Who Wins the Match Point in Tool Usage?

TL;DR

ACEBench tackles the lack of realistic, multi-turn tool-usage evaluation for LLMs by introducing Normal, Special, and Agent data categories and an automated sandboxed evaluation system. It covers 2,000 bilingual data entries across 8 domains and 68 sub-domains with 4,538 APIs, enabling end-to-end and granular diagnostics of tool invocation. Experimental results show closed-source models still lead, open-source models are catching up, and fine-tuning may harm generalization to imperfect instructions; Agent tasks remain the hardest benchmark. The framework reduces evaluation cost while providing deeper insights into tool-use capabilities and prompting strategies, with clear pathways for future expansion of scenarios and realism.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.
Paper Structure (36 sections, 4 equations, 35 figures, 8 tables)

This paper contains 36 sections, 4 equations, 35 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. ACEBench
  4. Dataset
  5. Data Construction
  6. Multi-Stage Data Verification
  7. Data Analysis
  8. Eval
  9. Normal Evaluation
  10. Special Evaluation
  11. Agent Evaluation
  12. Overall Accuracy
  13. Experiments
  14. Main results and analysis
  15. Error Analysis
  16. ...and 21 more sections

Figures (35)

  • Figure 1: Dataset construction pipeline. (a) Normal and Special data construction: API synthesis module (left), Dialogue generation module (right). (b) Agent Data Construction: include scenario construction, environment construction and question design.
  • Figure 2: Distribution of APIs in terms of domains (Top 2 subcategories for each category).
  • Figure 3: Visualization of the data composition of ACEBench.
  • Figure 4: Distribution of dialogue turns and API argument numbers.
  • Figure 5: Overview of evaluation process: The left represents 'Normal' evaluation: AST-based function and parameter verification. The middle illustrates 'Special' evaluation: Imperfect instruction defect diagnosis. The right shows 'Agent' evaluation: State transition analysis via user-model interaction
  • ...and 30 more figures