Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use

Fei Lei, Yibo Yang, Wenxiu Sun, Dahua Lin

TL;DR

MCPVerse introduces a real-world, large-scale benchmark for evaluating agentic tool use by LLMs, combining $65$ MCPs and $552$ tools to form an action space exceeding $140k$ tokens and employing time-sensitive ground truth. The framework uses outcome-based scoring across three modes (Oracle, Standard, Max-Scale) and a CAMEL-based evaluation pipeline to measure final task outcomes rather than prescribed tool sequences. Empirical results show that while many models degrade as tool space expands, agentic models like Claude-4-Sonnet and GLM-4.5 can exploit larger tool sets to improve performance, though overall success remains limited under Max-Scale constraints. MCPVerse provides a realistic, scalable benchmark for advancing agentic tool-use capabilities and motivates future work on scaling, evaluation pipelines, and robust tool orchestration.

Abstract

Large Language Models (LLMs) are evolving from text generators into reasoning agents. This transition makes their ability to use external tools a critical capability. However, evaluating this skill presents a significant challenge. Existing benchmarks are often limited by their reliance on synthetic tools and severely constrained action spaces. To address these limitations, we introduce MCPVerse, an expansive, real-world benchmark for evaluating agentic tool use. MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens, and employs outcome-based evaluation with real-time ground truth for time-sensitive tasks. We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale), revealing that while most models suffer performance degradation when confronted with larger tool sets, the agentic models, such as Claude-4-Sonnet, can effectively leverage expanded exploration spaces to improve accuracy. This finding not only exposes the limitations of state-of-the-art models in complex, real-world scenarios but also establishes MCPVerse as a critical benchmark for measuring and advancing agentic tool use capabilities.

MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use

TL;DR

MCPVerse introduces a real-world, large-scale benchmark for evaluating agentic tool use by LLMs, combining MCPs and tools to form an action space exceeding tokens and employing time-sensitive ground truth. The framework uses outcome-based scoring across three modes (Oracle, Standard, Max-Scale) and a CAMEL-based evaluation pipeline to measure final task outcomes rather than prescribed tool sequences. Empirical results show that while many models degrade as tool space expands, agentic models like Claude-4-Sonnet and GLM-4.5 can exploit larger tool sets to improve performance, though overall success remains limited under Max-Scale constraints. MCPVerse provides a realistic, scalable benchmark for advancing agentic tool-use capabilities and motivates future work on scaling, evaluation pipelines, and robust tool orchestration.

Abstract

Large Language Models (LLMs) are evolving from text generators into reasoning agents. This transition makes their ability to use external tools a critical capability. However, evaluating this skill presents a significant challenge. Existing benchmarks are often limited by their reliance on synthetic tools and severely constrained action spaces. To address these limitations, we introduce MCPVerse, an expansive, real-world benchmark for evaluating agentic tool use. MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens, and employs outcome-based evaluation with real-time ground truth for time-sensitive tasks. We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale), revealing that while most models suffer performance degradation when confronted with larger tool sets, the agentic models, such as Claude-4-Sonnet, can effectively leverage expanded exploration spaces to improve accuracy. This finding not only exposes the limitations of state-of-the-art models in complex, real-world scenarios but also establishes MCPVerse as a critical benchmark for measuring and advancing agentic tool use capabilities.

Paper Structure

This paper contains 26 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Tool Use
  4. Tool-Use Benchmarks
  5. MCPVerse Design
  6. MCP Collection
  7. Task Curation
  8. Evaluation Modes
  9. Evaluation Pipeline
  10. Task Formulation
  11. EXPERIMENT
  12. Setups
  13. Results Analysis
  14. Overall Model Performance
  15. The Impact of an Expanded Action Space
  16. ...and 11 more sections

Figures (4)

  • Figure 1: The Overview of MCPVerse. MCPVerse evaluation system. Tasks are curated with metadata and associated data files of various formats. Tools are collected from MCP hubs into an MCP pool and mounted as a toolset under three modes: Oracle, Standard, and Max-Scale. An LLM agent executes an agentic loop—planning and invoking tools via function calls—and produces an outcome. For time-invariant tasks the outcome is compared with human annotations; for time-sensitive tasks real-time scripts retrieve ground truth; an automated scorer then determines correctness.
  • Figure 2: Task distributions by (a) task type, (b) task complexity level, and (c) task time sensitivity.
  • Figure 3: Model performance accuracy across different evaluation modes.
  • Figure 4: Case study: Claude-4-Sonnet discovers a new solution path. Blocked by a strict authentication format in the limited Oracle mode, the model succeeds in Standard mode by pivoting to an alternative tool textttfetch from the expanded toolset.