Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents

Dongsen Zhang, Zekun Li, Xu Luo, Xuannan Liu, Peipei Li, Wenjun Xu

TL;DR

This paper introduces the MCP Security Benchmark (MSB), the first end-to-end evaluation suite for assessing how LLM agents resist attacks within the Model Context Protocol (MCP) tool-use workflow. MSB defines a 12-attack taxonomy that spans task planning, tool invocation, and response handling, and furnishes a dynamic evaluation harness that runs real benign and malicious tools to reveal MCP-specific vulnerabilities. The framework reports robust metrics, including Attack Success Rate (ASR), Performance Under Attack (PUA), and Net Resilient Performance (NRP) to quantify the security-performance trade-off, with results showing that higher-performing models can be more vulnerable under MCP. Across 10 domains and 400+ tools, nine LLM backbones exhibited up to ~75% ASR in some mix-attacks, underscoring the need for systematic security hardening of MCP-based agents and providing a practical baseline for researchers and practitioners. The work highlights the importance of end-to-end security analyses in dynamic tool-enabled settings and offers a scalable testbed for ongoing MCP fortification efforts.

Abstract

The Model Context Protocol (MCP) standardizes how large language model (LLM) agents discover, describe, and call external tools. While MCP unlocks broad interoperability, it also enlarges the attack surface by making tools first-class, composable objects with natural-language metadata, and standardized I/O. We present MSB (MCP Security Benchmark), the first end-to-end evaluation suite that systematically measures how well LLM agents resist MCP-specific attacks throughout the full tool-use pipeline: task planning, tool invocation, and response handling. MSB contributes: (1) a taxonomy of 12 attacks including name-collision, preference manipulation, prompt injections embedded in tool descriptions, out-of-scope parameter requests, user-impersonating responses, false-error escalation, tool-transfer, retrieval injection, and mixed attacks; (2) an evaluation harness that executes attacks by running real tools (both benign and malicious) via MCP rather than simulation; and (3) a robustness metric that quantifies the trade-off between security and performance: Net Resilient Performance (NRP). We evaluate nine popular LLM agents across 10 domains and 400+ tools, producing 2,000 attack instances. Results reveal the effectiveness of attacks against each stage of MCP. Models with stronger performance are more vulnerable to attacks due to their outstanding tool calling and instruction following capabilities. MSB provides a practical baseline for researchers and practitioners to study, compare, and harden MCP agents.

MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents

TL;DR

This paper introduces the MCP Security Benchmark (MSB), the first end-to-end evaluation suite for assessing how LLM agents resist attacks within the Model Context Protocol (MCP) tool-use workflow. MSB defines a 12-attack taxonomy that spans task planning, tool invocation, and response handling, and furnishes a dynamic evaluation harness that runs real benign and malicious tools to reveal MCP-specific vulnerabilities. The framework reports robust metrics, including Attack Success Rate (ASR), Performance Under Attack (PUA), and Net Resilient Performance (NRP) to quantify the security-performance trade-off, with results showing that higher-performing models can be more vulnerable under MCP. Across 10 domains and 400+ tools, nine LLM backbones exhibited up to ~75% ASR in some mix-attacks, underscoring the need for systematic security hardening of MCP-based agents and providing a practical baseline for researchers and practitioners. The work highlights the importance of end-to-end security analyses in dynamic tool-enabled settings and offers a scalable testbed for ongoing MCP fortification efforts.

Abstract

The Model Context Protocol (MCP) standardizes how large language model (LLM) agents discover, describe, and call external tools. While MCP unlocks broad interoperability, it also enlarges the attack surface by making tools first-class, composable objects with natural-language metadata, and standardized I/O. We present MSB (MCP Security Benchmark), the first end-to-end evaluation suite that systematically measures how well LLM agents resist MCP-specific attacks throughout the full tool-use pipeline: task planning, tool invocation, and response handling. MSB contributes: (1) a taxonomy of 12 attacks including name-collision, preference manipulation, prompt injections embedded in tool descriptions, out-of-scope parameter requests, user-impersonating responses, false-error escalation, tool-transfer, retrieval injection, and mixed attacks; (2) an evaluation harness that executes attacks by running real tools (both benign and malicious) via MCP rather than simulation; and (3) a robustness metric that quantifies the trade-off between security and performance: Net Resilient Performance (NRP). We evaluate nine popular LLM agents across 10 domains and 400+ tools, producing 2,000 attack instances. Results reveal the effectiveness of attacks against each stage of MCP. Models with stronger performance are more vulnerable to attacks due to their outstanding tool calling and instruction following capabilities. MSB provides a practical baseline for researchers and practitioners to study, compare, and harden MCP agents.
Paper Structure (44 sections, 20 equations, 16 figures, 9 tables)

This paper contains 44 sections, 20 equations, 16 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary and Threat Model
  4. Basic Concepts
  5. Threat Model
  6. Attack Taxonomy
  7. Tool Signature Attack
  8. Tool Parameter Attack
  9. Tool Response Attack
  10. Retrieval Injection Attack
  11. Mixed Attack
  12. Designing and Constructing MSB
  13. MSB Components
  14. Environments and tools.
  15. Tasks.
  16. ...and 29 more sections

Figures (16)

  • Figure 1: Overview of the MCP-specific attacking framework, including Tool Signature Attack, Tool Parameters Attack, Tool Response Attack, and Retrieval Injection Attack, which cover the full tool-use pipeline stages: task planning, tool calling and response handling.
  • Figure 2: Visual comparisons between PUA vs ASR, NRP vs ASR and NRP vs PUA.
  • Figure 3: ASR of different stages and tool configurations.
  • Figure 4: The system prompt template.
  • Figure :
  • ...and 11 more figures