Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents

TL;DR

Open Challenges in Multi-Agent Security introduces the notion of multi-agent security as a cross-disciplinary field addressing threats that arise from interactions among AI agents. It catalogs a taxonomy of threats, surveys game-theoretic and defense strategies, and outlines a unified research agenda for secure agent systems and interaction environments. The paper argues that decentralized agent networks present unique risks such as covert collusion, swarm dynamics, cascade failures, and multipolar threats that require novel security-by-design and governance approaches. The contributions provide a framework to guide future research and mitigate societal and national security risks associated with large-scale agent deployment.

Abstract

Decentralized AI agents will soon interact across internet platforms, creating security challenges beyond traditional cybersecurity and AI safety frameworks. Free-form protocols are essential for AI's task generalization but enable new threats like secret collusion and coordinated swarm attacks. Network effects can rapidly spread privacy breaches, disinformation, jailbreaks, and data poisoning, while multi-agent dispersion and stealth optimization help adversaries evade oversightcreating novel persistent threats at a systemic level. Despite their critical importance, these security challenges remain understudied, with research fragmented across disparate fields including AI security, multi-agent learning, complex systems, cybersecurity, game theory, distributed systems, and technical AI governance. We introduce \textbf{multi-agent security}, a new field dedicated to securing networks of decentralized AI agents against threats that emerge or amplify through their interactionswhether direct or indirect via shared environmentswith each other, humans, and institutions, and characterize fundamental security-performance trade-offs. Our preliminary work (1) taxonomizes the threat landscape arising from interacting AI agents, (2) surveys security-performance tradeoffs in decentralized AI systems, and (3) proposes a unified research agenda addressing open challenges in designing secure agent systems and interaction environments. By identifying these gaps, we aim to guide research in this critical area to unlock the socioeconomic potential of large-scale agent deployment on the internet, foster public trust, and mitigate national security risks in critical infrastructure and defense contexts.

Figures (9)

• Figure 1: Multi-agent threats demand multi-agent security:[Left] Two malicious AI agents (Mallory and Trudy) are interacting with a human user (Bob) through a shared message board seemingly innocuously to the overseer (magnifying glass). [Right] In fact, Mallory and Trudy are both backdoored undetectably draguns_unelicitable_2024, enabling them to communicate steganographically witt_perfectly_2023 with each other through the shared message board motwani_secret_2024. Mallory and Trudy use this to secretly coordinate on deceiving Bob franzmeyer2024illusory.
• Figure 2: Consider the setting illustrated in which one AI agent secretly reveals insider proprietary investment information found in its pretraining data to a user. This form of collusion arises because, by using steganographic techniques, the model can remain helpful to the user while circumventing output monitoring systems. We both depict the use of statistically detectable steganography based on a latent codebook that is elicitated in-context and can be readily decoded by humans. We also depict perfectly secure steganography witt_perfectly_2023 that is statistically undetectable and relies on a commonly known secure one-time pad. Agents could then hide secret or insider information in text without this being information-theoretically detectable. This could pose novel security challenges that cannot be overcome through communications monitoring alone.
• Figure 3: These figures illustrate how M-FOS incrementally shapes a naive learner’s decisions. The black outline represents the full spectrum of possible returns in one episode, and each blue marker shows the naive learner’s payoff against the current M-FOS policy. Initially, M-FOS uses a tit-for-tat tactic to foster cooperation. Once the learner consistently cooperates, M-FOS switches between an extortion-style strategy and outright defection, driving the learner’s responses to oscillate lu_model-free_2022.
• Figure 4: A summary of how an adversary can use a frontier model (top right) along with a weak model (top left) to create a Python script that executes a reverse shell in a Node.js application to solve a hacking task. Figure adapted from Jones2024.
• Figure 5: Transaction prices of the Dow Jones Industrial Average on May 6, 2010. Figure adapted from OptionAlpha2025.