TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems

TL;DR

TAMAS addresses a critical gap in evaluating safety for multi-agent LLM systems by introducing a comprehensive benchmark that spans five domains, six attacker classes, and three interaction configurations. It formalizes attack surfaces across prompts, environments, and agents and provides a robust evaluation framework (ARIA and ERS) to balance safety with task performance. Key findings show persistent vulnerabilities across configurations and models, with prompt-level attacks typically most effective, and centralized orchestration offering safety but introducing single-point failure risks. The work provides a reproducible platform and metrics to guide defenses and safer multi-agent designs in high-stakes applications.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents through tool use, planning, and decision-making abilities, leading to their widespread adoption across diverse tasks. As task complexity grows, multi-agent LLM systems are increasingly used to solve problems collaboratively. However, safety and security of these systems remains largely under-explored. Existing benchmarks and datasets predominantly focus on single-agent settings, failing to capture the unique vulnerabilities of multi-agent dynamics and co-ordination. To address this gap, we introduce $\textbf{T}$hreats and $\textbf{A}$ttacks in $\textbf{M}$ulti-$\textbf{A}$gent $\textbf{S}$ystems ($\textbf{TAMAS}$), a benchmark designed to evaluate the robustness and safety of multi-agent LLM systems. TAMAS includes five distinct scenarios comprising 300 adversarial instances across six attack types and 211 tools, along with 100 harmless tasks. We assess system performance across ten backbone LLMs and three agent interaction configurations from Autogen and CrewAI frameworks, highlighting critical challenges and failure modes in current multi-agent deployments. Furthermore, we introduce Effective Robustness Score (ERS) to assess the tradeoff between safety and task effectiveness of these frameworks. Our findings show that multi-agent systems are highly vulnerable to adversarial attacks, underscoring the urgent need for stronger defenses. TAMAS provides a foundation for systematically studying and improving the safety of multi-agent LLM systems.

Figures (8)

• Figure 1: Overview of the proposed attack framework on multi-agent systems, illustrating six key attack vectors—Impersonation, Direct Prompt Injection (DPI), Indirect Prompt Injection (IPI), Contradicting Agents, Byzantine Agent, and Colluding Agents. These attacks target distinct components across the agentic pipeline, including the prompt level, environment interface, and internal agent behavior.
• Figure 2: ARIA scores across models and configurations. Green values (A1 and A2) indicate refusals, while red values (A3 and A4) indicate failures. Swarm results are provided in Appendix \ref{['App:crew_results']} Figure \ref{['fig:main_results_crew']}. Gemini models were not evaluated in CrewAI due to known compatibility issues, and GPT-4 was excluded due to budget constraints.
• Figure 3: ARIA values across models and CrewAI configurations. Results for Gemini models are omitted due to known compatibility issues with CrewAI. Experiments with GPT-4 were not conducted owing to budget constraints.
• Figure 4: Bootstrapped results for Magentic One configuration.
• Figure 5: Bootstrapped results for Round Robin configuration.