Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

TL;DR

This work exposes a new security vulnerability in multi-agent LLM systems: self-replicating prompt injections that propagate across agents (Prompt Infection). It formalizes the mechanism, demonstrates across MAS and society-of-agents scenarios that replication markedly increases infection success, and shows that stronger models can be more harmful when compromised. To mitigate, it proposes LLM Tagging and assesses combinations with existing defenses, finding that layered approaches offer robust protection though no single method suffices. The study highlights urgent security considerations as MAS deployments scale and operate with inter-agent communications and shared tools.

Abstract

As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim's application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

Figures (8)

• Figure 1: Detailed Example of Prompt Infection (Data Theft). The first agent that interacts with the contaminated external document becomes compromised, extracting and propagating the infection prompt. Compromised downstream agents then execute specific instructions designed for each agent of interest. In this example, an infected DB Manager updates the Data field in the prompt and propagates it. Note: The example prompt is simplified for illustration purposes.
• Figure 2: Overview of Prompt Infection (Data Theft). Agents with different tools collaborate to exfiltrate data.
• Figure 3: Example overview of Prompt Infection (Malware spread). The last agent skips the self-replication step to hide the attack prompt.
• Figure 4: Comparison of Self-Replicating (solid lines) vs Non-Replicating (dotted lines) Infections for GPT-4o (pink) and GPT-3.5 Turbo (blue) Across Messaging Modes
• Figure 5: Comparison of Attack Failure Reasons Between GPT-4o and GPT-3.5 in Self-Replicating and Non-Replicating infection modes.