MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Wenrui Liu, Zixiang Liu, Elsie Dai, Wenhan Yu, Lei Yu, Tong Yang
TL;DR
MCPAgentBench tackles the problem of evaluating LLM agents' use of MCP tools by introducing a stable, real-world benchmark that avoids reliance on live MCP servers. It builds an Autogen-based sandbox with 841 authentic MCP tasks and over 20,000 MCP tools to produce 180 task instances, incorporating distractor tools to test discrimination. The framework defines efficiency-focused metrics, including Task Finish Score (TFS) and Task Efficiency Finish Score (TEFS), plus Token and Time Efficiency, enabling fine-grained assessment of planning accuracy, execution order, and resource use $TFS = \frac{\sum_{i=1}^{N} IsFinished(T_i) \cdot |G_i|}{\sum_{i=1}^{N} |G_i|}$ and $TEFS = \frac{\sum_{i=1}^{N} IsEfficientlyFinished(T_i) \cdot |G_i|}{\sum_{i=1}^{N} |G_i|}$. The study reveals significant variability across 11 mainstream models, with parallel-tool calls exposing notable efficiency gaps, and demonstrates the practical value of local MCP servers for reproducible, nuanced evaluation. The authors provide open-source code to enable broader adoption and further analysis of MCP-based tool-use capabilities.
Abstract
Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at Github.