Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents

TL;DR

The paper tackles the lack of diagnostic insight in tool-using LLM benchmarks by introducing a Cognitive Load Theory–based framework that decomposes task difficulty into Intrinsic Load ($CL_I$) and Extraneous Load ($CL_E$) via a Tool Interaction Graph (TIG) and a parametric benchmark called ToolLoad-Bench. It derives an additive load model and a predictive accuracy relation $\text{Accuracy} \approx \exp(-(k \cdot CL_{Total} + b))$ with $CL_{Total}=CL_I+\omega_E\,CL_E$, and validates predictions through calibration tests such as the Hosmer-Lemeshow test. The work maps model capability boundaries, reveals sharp performance cliffs under increasing cognitive load, and demonstrates practical utility for intelligent task routing and robust evaluation of tool-use agents. Overall, the framework provides a principled, diagnostic alternative to single-score benchmarks, enabling targeted improvements and deployment strategies for tool-augmented systems.

Abstract

The ability of Large Language Models (LLMs) to use external tools unlocks powerful real-world interactions, making rigorous evaluation essential. However, current benchmarks primarily report final accuracy, revealing what models can do but obscuring the cognitive bottlenecks that define their true capability boundaries. To move from simple performance scoring to a diagnostic tool, we introduce a framework grounded in Cognitive Load Theory. Our framework deconstructs task complexity into two quantifiable components: Intrinsic Load, the inherent structural complexity of the solution path, formalized with a novel Tool Interaction Graph; and Extraneous Load, the difficulty arising from ambiguous task presentation. To enable controlled experiments, we construct ToolLoad-Bench, the first benchmark with parametrically adjustable cognitive load. Our evaluation reveals distinct performance cliffs as cognitive load increases, allowing us to precisely map each model's capability boundary. We validate that our framework's predictions are highly calibrated with empirical results, establishing a principled methodology for understanding an agent's limits and a practical foundation for building more efficient systems.

Figures (6)

• Figure 1: An illustration of our Cognitive Load Framework. A multi-turn tool-use task, defined by a sequence of user queries and a set of tools , is mapped to a TIG. The TIG represents the ground-truth solution. Zooming in node level, It shows selection load in parameter extraction and extraneous cognitive Load in tool selection.
• Figure 2: Accuracy vs. Intrinsic Cognitive Load ($CL_I$).
• Figure 3: Accuracy vs. Extraneous Cognitive Load ($CL_E$).
• Figure 4: Boxplots showing the distribution of Total Cognitive Load ($CL_{Total}$) for successfully completed tasks by different models.
• Figure 5: Empirical accuracy vs. Total Cognitive Load. The blue bars show the actual accuracy within binned load intervals, while the red line represents the fitted exponential decay curve from our theoretical model.