Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu

TL;DR

MCP-Atlas provides a large-scale, real-server benchmark to evaluate tool-use competency for MCP-enabled LLM agents. It assembles 36 real MCP servers and 220 tools into 1,000 tasks that require 3–6 tool calls across multiple servers, with prompts that avoid naming tools to stress discovery and orchestration. A claims-based scoring rubric, supplemented by internal diagnostics, enables objective partial-credit evaluation and diagnostics on tool discovery, parameterization, and recovery. Frontier models reach up to 62.3% pass rate, signaling progress but also substantial room for improvement in multi-step, real-world tool use; the authors release the harness and a public 500-task subset to foster reproducible, comparative research.

Abstract

The Model Context Protocol (MCP) is rapidly becoming the standard interface for Large Language Models (LLMs) to discover and invoke external tools. However, existing evaluations often fail to capture the complexity of real-world scenarios, relying on restricted toolsets, simplistic workflows, or subjective LLM-as-a-judge metrics. We introduce MCP-Atlas, a large-scale benchmark for evaluating tool-use competency, comprising 36 real MCP servers and 220 tools. It includes 1,000 tasks designed to assess tool-use competency in realistic, multi-step workflows. Tasks use natural language prompts that avoid naming specific tools or servers, requiring agents to identify and orchestrate 3-6 tool calls across multiple servers. We score tasks using a claims-based rubric that awards partial credit based on the factual claims satisfied in the model's final answer, complemented by internal diagnostics on tool discovery, parameterization, syntax, error recovery, and efficiency. Evaluation results on frontier models reveal that top models achieve pass rates exceeding 50%, with primary failures arising from inadequate tool usage and task understanding. We release the task schema, containerized harness, and a 500-task public subset of the benchmark dataset to facilitate reproducible comparisons and advance the development of robust, tool-augmented agents.

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

TL;DR

MCP-Atlas provides a large-scale, real-server benchmark to evaluate tool-use competency for MCP-enabled LLM agents. It assembles 36 real MCP servers and 220 tools into 1,000 tasks that require 3–6 tool calls across multiple servers, with prompts that avoid naming tools to stress discovery and orchestration. A claims-based scoring rubric, supplemented by internal diagnostics, enables objective partial-credit evaluation and diagnostics on tool discovery, parameterization, and recovery. Frontier models reach up to 62.3% pass rate, signaling progress but also substantial room for improvement in multi-step, real-world tool use; the authors release the harness and a public 500-task subset to foster reproducible, comparative research.

Abstract

The Model Context Protocol (MCP) is rapidly becoming the standard interface for Large Language Models (LLMs) to discover and invoke external tools. However, existing evaluations often fail to capture the complexity of real-world scenarios, relying on restricted toolsets, simplistic workflows, or subjective LLM-as-a-judge metrics. We introduce MCP-Atlas, a large-scale benchmark for evaluating tool-use competency, comprising 36 real MCP servers and 220 tools. It includes 1,000 tasks designed to assess tool-use competency in realistic, multi-step workflows. Tasks use natural language prompts that avoid naming specific tools or servers, requiring agents to identify and orchestrate 3-6 tool calls across multiple servers. We score tasks using a claims-based rubric that awards partial credit based on the factual claims satisfied in the model's final answer, complemented by internal diagnostics on tool discovery, parameterization, syntax, error recovery, and efficiency. Evaluation results on frontier models reveal that top models achieve pass rates exceeding 50%, with primary failures arising from inadequate tool usage and task understanding. We release the task schema, containerized harness, and a 500-task public subset of the benchmark dataset to facilitate reproducible comparisons and advance the development of robust, tool-augmented agents.
Paper Structure (42 sections, 3 equations, 2 figures, 11 tables)

This paper contains 42 sections, 3 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Evolution from Static to Agentic Evaluation
  4. MCP-Based Evaluation
  5. Benchmark Design and Overview
  6. Scope and Scale
  7. Task Schema
  8. Exposure and Distractor Policy.
  9. Environment Buckets and Taxonomy
  10. Quality Assurance
  11. Evaluation Methodology and Experimental Setup
  12. Evaluation Protocol
  13. Judge Model and Human Alignment.
  14. Diagnostics
  15. Environment and Harness
  16. ...and 27 more sections

Figures (2)

  • Figure 1: MCP-Atlas overall model performance ranked by pass rate on the 1,000 task set. The best-performing model (Claude Opus 4.5) achieves 62.3% success, highlighting progress while showing room for improvement on multi-step, real-world tool use tasks.
  • Figure 2: Diagnostic categories for the top 3 models on failed tasks. The y-axis shows average coverage score across all models (i.e. task difficulty, higher average coverage score indicates lower task difficulty), while the x-axis shows model score. Colors indicate failure reasons, classified by an LLM judge with access to the ground truth and generated trajectory.