Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems

Xu He, Di Wu, Yan Zhai, Kun Sun

TL;DR

This work addresses security and reliability challenges in LLM-based multi-agent systems by proposing a topology-agnostic Graph-Based Anomaly Detection Framework and a runtime SentinelAgent. It models MAS executions as dynamic interaction graphs with nodes representing agents/tools and edges representing communications and invocations, enabling node-, edge-, and path-level anomaly reasoning. The SentinelAgent uses an LLM-driven 'judge' plus human-in-the-loop policy refinement to detect, attribute, and intervene on global, single-point, and multi-point failures. Case studies on an automated email assistant and Microsoft's Magentic-One demonstrate detection of prompt injections, unsafe tool usage, and inter-agent collusion with explainable root-cause traces. The approach offers scalable, explainable security for diverse MAS deployments and points to future work in policy learning and formal verification.

Abstract

The rise of large language model (LLM)-based multi-agent systems (MAS) introduces new security and reliability challenges. While these systems show great promise in decomposing and coordinating complex tasks, they also face multi-faceted risks across prompt manipulation, unsafe tool usage, and emergent agent miscoordination. Existing guardrail mechanisms offer only partial protection, primarily at the input-output level, and fall short in addressing systemic or multi-point failures in MAS. In this work, we present a system-level anomaly detection framework tailored for MAS, integrating structural modeling with runtime behavioral oversight. Our approach consists of two components. First, we propose a graph-based framework that models agent interactions as dynamic execution graphs, enabling semantic anomaly detection at node, edge, and path levels. Second, we introduce a pluggable SentinelAgent, an LLM-powered oversight agent that observes, analyzes, and intervenes in MAS execution based on security policies and contextual reasoning. By bridging abstract detection logic with actionable enforcement, our method detects not only single-point faults and prompt injections but also multi-agent collusion and latent exploit paths. We validate our framework through two case studies, including an email assistant and Microsoft's Magentic-One system, demonstrating its ability to detect covert risks and provide explainable root-cause attribution. Our work lays the foundation for more trustworthy, monitorable, and secure agent-based AI ecosystems.

SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems

TL;DR

This work addresses security and reliability challenges in LLM-based multi-agent systems by proposing a topology-agnostic Graph-Based Anomaly Detection Framework and a runtime SentinelAgent. It models MAS executions as dynamic interaction graphs with nodes representing agents/tools and edges representing communications and invocations, enabling node-, edge-, and path-level anomaly reasoning. The SentinelAgent uses an LLM-driven 'judge' plus human-in-the-loop policy refinement to detect, attribute, and intervene on global, single-point, and multi-point failures. Case studies on an automated email assistant and Microsoft's Magentic-One demonstrate detection of prompt injections, unsafe tool usage, and inter-agent collusion with explainable root-cause traces. The approach offers scalable, explainable security for diverse MAS deployments and points to future work in policy learning and formal verification.

Abstract

The rise of large language model (LLM)-based multi-agent systems (MAS) introduces new security and reliability challenges. While these systems show great promise in decomposing and coordinating complex tasks, they also face multi-faceted risks across prompt manipulation, unsafe tool usage, and emergent agent miscoordination. Existing guardrail mechanisms offer only partial protection, primarily at the input-output level, and fall short in addressing systemic or multi-point failures in MAS. In this work, we present a system-level anomaly detection framework tailored for MAS, integrating structural modeling with runtime behavioral oversight. Our approach consists of two components. First, we propose a graph-based framework that models agent interactions as dynamic execution graphs, enabling semantic anomaly detection at node, edge, and path levels. Second, we introduce a pluggable SentinelAgent, an LLM-powered oversight agent that observes, analyzes, and intervenes in MAS execution based on security policies and contextual reasoning. By bridging abstract detection logic with actionable enforcement, our method detects not only single-point faults and prompt injections but also multi-agent collusion and latent exploit paths. We validate our framework through two case studies, including an email assistant and Microsoft's Magentic-One system, demonstrating its ability to detect covert risks and provide explainable root-cause attribution. Our work lays the foundation for more trustworthy, monitorable, and secure agent-based AI ecosystems.

Paper Structure

This paper contains 34 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Threat Landscape and Problem Formulation
  3. Risk Taxonomy in Multi-Agent Systems
  4. Threat Model
  5. Multi-Tier Anomaly Detection Objectives
  6. Multi-Agent System Topologies and Their Implications
  7. Round Robin
  8. Central Orchestrator
  9. Central Orchestrator with External Ledger
  10. Graph-Based Anomaly Detection Framework
  11. Graph Modeling of MAS
  12. Attack Path-Based Failure Attribution
  13. Implementation of the Graph-Based Approach
  14. SentinelAgent for Runtime Oversight
  15. SentinelAgent Architecture
  16. ...and 19 more sections

Figures (6)

  • Figure 1: Risk types in Multi-Agent System
  • Figure 2: Representative topologies in MAS: (a) Round Robin, (b) Central Orchestrator, (c) Central Orchestrator with External Ledger.
  • Figure 3: Interaction Graph of MAS
  • Figure 4: The interaction between SentinelAgent with MAS.
  • Figure 5: System architecture of the email assistant multi-agent system. Agent and tool nodes form a directed interaction graph with clearly defined execution pathways.
  • ...and 1 more figures