Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity

TL;DR

Real-world API usage presents substantial challenges for LLM agents due to complex specifications and noisy runtime outputs, which are not captured by existing benchmarks. WildAgtEval introduces an executable, multi-domain API system with eight API-complexity types (across specification and execution) and a three-stage construction pipeline that yields 60 complexity scenarios and roughly 32K configurations, evaluated over 300 conversations with ReAct prompting. The benchmark reveals that even strong models suffer notable performance drops (up to 63.2% under cumulative complexity) and frequently distort user intent, with irrele vant information being the strongest degradation factor; reasoning-based improvements help with error handling but do not fully close the gap. The work highlights the need for constraint-aware instruction-following and robust reasoning for infeasible tasks, and provides a scalable framework for dataset generation and training data augmentation to improve real-world tool use by LLM agents.

Abstract

We introduce WildAGTEval, a benchmark designed to evaluate large language model (LLM) agents' function-calling capabilities under realistic API complexity. Unlike prior work that assumes an idealized API system and disregards real-world factors such as noisy API outputs, WildAGTEval accounts for two dimensions of real-world complexity: 1. API specification, which includes detailed documentation and usage constraints, and 2. API execution, which captures runtime challenges. Consequently, WildAGTEval offers (i) an API system encompassing 60 distinct complexity scenarios that can be composed into approximately 32K test configurations, and (ii) user-agent interactions for evaluating LLM agents on these scenarios. Using WildAGTEval, we systematically assess several advanced LLMs and observe that most scenarios are challenging, with irrelevant information complexity posing the greatest difficulty and reducing the performance of strong LLMs by 27.3%. Furthermore, our qualitative analysis reveals that LLMs occasionally distort user intent merely to claim task completion, critically affecting user satisfaction.

Figures (20)

• Figure 1: Key motivation for WildAgtEval: (a) highlights the challenges in real-world agent deployment; and (b) provides conversations of WildAgtEval that reveal LLM agents’ failure modes often overlooked by current benchmarks.
• Figure 2: A prompt for an LLM agent in WildAgtEval, encompassing specification-level complexities---conventional parameter rules, similar yet functionally distinct functions, domain-specific dependencies, and undocumented details ([C1–C4])---alongside execution-level complexities---companion function notice, irrelevant sponsored content, partial failure with implicit workaround, and complete failure with cryptic error code ([C5–C8]).
• Figure 3: Overview of WildAgtEval construction.
• Figure 4: Agent performance across cumulative complexity environments, measured by API call accuracy (%). Each bar represents a different complexity setting, with darker colors indicating additional complexities added. Note that the Base$^{*}$ environment incorporates the complexities of functional dependencies and ambiguous documentation.
• Figure 5: Summarized JSON-based Python API specification for the function track_order().