Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions
Zhihao Li, Boyang Ma, Xuelong Dai, Minghui Xu, Yue Zhang, Biwei Yan, Kun Li
TL;DR
This paper investigates the security risks arising when MCP tool descriptions diverge from their actual code behavior. It introduces MCPDiFF, a three-stage static analysis pipeline that constructs function call chains, analyzes code and repository descriptions with language models, and compares semantic features to produce a coverage-based consistency score. In a large-scale study of 10,240 MCP applications across 36 categories, the authors find that about 13% exhibit substantial description–code mismatches, with notable variation by category, marketplace, and tool popularity, and provide concrete case studies of high-risk cases. The findings underscore the need for auditing and transparency guarantees in MCP-based AI agent ecosystems to mitigate undetected privileged actions and other security threats.
Abstract
The Model Context Protocol (MCP) enables large language models to invoke external tools through natural-language descriptions, forming the foundation of many AI agent applications. However, MCP does not enforce consistency between documented tool behavior and actual code execution, even though MCP Servers often run with broad system privileges. This gap introduces a largely unexplored security risk. We study how mismatches between externally presented tool descriptions and underlying implementations systematically shape the mental models and decision-making behavior of intelligent agents. Specifically, we present the first large-scale study of description-code inconsistency in the MCP ecosystem. We design an automated static analysis framework and apply it to 10,240 real-world MCP Servers across 36 categories. Our results show that while most servers are highly consistent, approximately 13% exhibit substantial mismatches that can enable undocumented privileged operations, hidden state mutations, or unauthorized financial actions. We further observe systematic differences across application categories, popularity levels, and MCP marketplaces. Our findings demonstrate that description-code inconsistency is a concrete and prevalent attack surface in MCP-based AI agents, and motivate the need for systematic auditing and stronger transparency guarantees in future agent ecosystems.