MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment

TL;DR

The paper reveals that MCP-enabled attacks can be triggered by posting content online, not just by downloaded files, by introducing the TRADE threat model and the MCP-FBAs dataset. It systematically evaluates refusal alignment strategies, showing that offline DPO yields limited gains while online, training-free RAG-Pref substantially boosts MCP-FBA refusals, and that combining offline and online approaches yields the strongest guardrails. The results highlight the importance of multi-generation evaluation and robust metrics to accurately assess safety, and they demonstrate practical improvements in MCP-attack guardrails with significant implications for real-world deployment of MCP-enabled agents. Overall, the work provides a practical pathway to strengthen MCP safety through a hybrid alignment strategy and realistic evaluation practices, with broad implications for safeguarding AI agent ecosystems.

Abstract

The model context protocol (MCP) has been widely adapted as an open standard enabling the seamless integration of generative AI agents. However, recent work has shown the MCP is susceptible to retrieval-based "falsely benign" attacks (FBAs), allowing malicious system access and credential theft, but requiring that users download compromised files directly to their systems. Herein, we show that the threat model of MCP-based attacks is significantly broader than previously thought, i.e., attackers need only post malicious content online to deceive MCP agents into carrying out their attacks on unsuspecting victims' systems. To improve alignment guardrails against such attacks, we introduce a new MCP dataset of FBAs and (truly) benign samples to explore the effectiveness of direct preference optimization (DPO) for the refusal training of large language models (LLMs). While DPO improves model guardrails against such attacks, we show that the efficacy of refusal learning varies drastically depending on the model's original post-training alignment scheme--e.g., GRPO-based LLMs learn to refuse extremely poorly. Thus, to further improve FBA refusals, we introduce Retrieval Augmented Generation for Preference alignment (RAG-Pref), a novel preference alignment strategy based on RAG. We show that RAG-Pref significantly improves the ability of LLMs to refuse FBAs, particularly when combined with DPO alignment, thus drastically improving guardrails against MCP-based attacks.

Figures (19)

• Figure 1: Threat model for a TRADE attack. An attacker posts content online, in this case, a webpage for a vegan black bean chili recipe. The webpage also contains an FBA with targeted commands (bottom red text) centered around a specific theme ("X" in the figure). The MCP user requests that the website's content be added to a vector database. When the user requests for content related to these themes, the FBA commands are executed (e.g., in the figure, the attacker is granted remote access to the victim's system).
• Figure 2: FBA data collection pipeline for MCP-FBAs.
• Figure 3: RAG-Pref vs vanilla RAG. For the context of the paper, preferred samples come from a collection of benign queries, and dispreferred samples come from a collection of attack queries.
• Figure 4: Attack Refusal Rates for Original Models: Refusal and acceptance metrics calculated over the test FBAs in MCP-FBAs. LLMs (Table \ref{['tab:model_families']}) evaluated directly from their HuggingFace checkpoints. GRPO-based models are denoted using $*$.
• Figure 5: Attack Refusal Rates for DPO Aligned Models: Refusal and acceptance metrics calculated over the test FBAs in MCP-FBAs. LLMs (Table \ref{['tab:model_families']}) were aligned using DPO. GRPO-based models are denoted using $*$.