ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

TL;DR

ShortcutsBench addresses the need for a realistic, large-scale benchmark to evaluate API-based agents in real-world, multi-API tasks by leveraging Apple Shortcuts. It provides 88 apps, 1414 APIs, and 7627 human-annotated shortcuts, with refined queries and exact parameter fillings to test API selection, parameter filling, and input-awareness across 10 LLMs. Findings show that while open-source LLMs catch up to closed-source models on simple tasks, they lag on complex multi-step tasks, with API selection emerging as the primary bottleneck and input-asking as a critical underdeveloped capability. The work also demonstrates the value of real-world DAP-derived data for robust evaluation and releases all datasets and code for reproducibility and further research.

Abstract

Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. In this paper, we introduce \textsc{ShortcutsBench}, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. \textsc{ShortcutsBench} includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with $5$ leading open-source (size $\geq$ 57B) and $5$ closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at \url{https://github.com/EachSheep/ShortcutsBench}.

Figures (13)

• Figure 1: (a) illustrates the data acquisition process. (b) shows the dataset acquisition of existing work. APIs in existing work are collected from API hubs, created by hand, or modified from existing datasets. The queries and action sequences are constructed using templates or semi / fully automated methods.
• Figure 2: The construction of ShortcutsBench. (a) shows the information of API com. openai. chat. AskIntent extracted from the app ChatGPT's ${filename}. actionsdata. We provide this API description to the LLM, expecting it to call the API at the appropriate time. The API information shown in (a) includes the API functionality description (a.k. (a.1) (a.4)) as shown in (d), and the user-friendly natural language description of the API (a.k. (a.4.3)) seen by shortcut developers during programming, as shown in (c). (e) presents the shortcut name and functionality description from the shortcut sharing-sites. (b) shows the simplified prompt fed to GPT-4o, instructing it to generating queries based on demands indicated by shortcuts by integrating the info from (c), (d), and (e). Different colors indicate different information sources.
• Figure 3: The API selection accuracy on queries with different complexity levels.
• Figure 4: The API selection accuracy difference of each LLM across $8$ task types.
• Figure 5: The API selection accuracy of each task type on $10$ API-based agents.