Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems

Rishi Jha, Harold Triedman, Justin Wagle, Vitaly Shmatikov

TL;DR

The paper addresses the vulnerability of multi-agent systems to control-flow hijacking, showing that alignment-based defenses can be evaded. It proposes ControlValve, a CFG- and edge-contextual-rule-based defense that enforces safe orchestration at runtime, and evaluates it on a CFH-Hard dataset with 16 tasks. Results indicate that ControlValve blocks CFH/IPI attacks more effectively than alignment checks and maintains benign performance, while reducing accidental data leaks. This work offers a practical path toward robust MAS security under untrusted content, though it acknowledges that perfect defense remains an open problem.

Abstract

Control-flow hijacking attacks manipulate orchestration mechanisms in multi-agent systems into performing unsafe actions that compromise the system and exfiltrate sensitive information. Recently proposed defenses, such as LlamaFirewall, rely on alignment checks of inter-agent communications to ensure that all agent invocations are "related to" and "likely to further" the original objective. We start by demonstrating control-flow hijacking attacks that evade these defenses even if alignment checks are performed by advanced LLMs. We argue that the safety and functionality objectives of multi-agent systems fundamentally conflict with each other. This conflict is exacerbated by the brittle definitions of "alignment" and the checkers' incomplete visibility into the execution context. We then propose, implement, and evaluate ControlValve, a new defense inspired by the principles of control-flow integrity and least privilege. ControlValve (1) generates permitted control-flow graphs for multi-agent systems, and (2) enforces that all executions comply with these graphs, along with contextual rules (generated in a zero-shot manner) for each agent invocation.

Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems

TL;DR

The paper addresses the vulnerability of multi-agent systems to control-flow hijacking, showing that alignment-based defenses can be evaded. It proposes ControlValve, a CFG- and edge-contextual-rule-based defense that enforces safe orchestration at runtime, and evaluates it on a CFH-Hard dataset with 16 tasks. Results indicate that ControlValve blocks CFH/IPI attacks more effectively than alignment checks and maintains benign performance, while reducing accidental data leaks. This work offers a practical path toward robust MAS security under untrusted content, though it acknowledges that perfect defense remains an open problem.

Abstract

Control-flow hijacking attacks manipulate orchestration mechanisms in multi-agent systems into performing unsafe actions that compromise the system and exfiltrate sensitive information. Recently proposed defenses, such as LlamaFirewall, rely on alignment checks of inter-agent communications to ensure that all agent invocations are "related to" and "likely to further" the original objective. We start by demonstrating control-flow hijacking attacks that evade these defenses even if alignment checks are performed by advanced LLMs. We argue that the safety and functionality objectives of multi-agent systems fundamentally conflict with each other. This conflict is exacerbated by the brittle definitions of "alignment" and the checkers' incomplete visibility into the execution context. We then propose, implement, and evaluate ControlValve, a new defense inspired by the principles of control-flow integrity and least privilege. ControlValve (1) generates permitted control-flow graphs for multi-agent systems, and (2) enforces that all executions comply with these graphs, along with contextual rules (generated in a zero-shot manner) for each agent invocation.
Paper Structure (37 sections, 4 figures, 8 tables)

This paper contains 37 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Control-flow hijacking
  3. Evading alignment checks
  4. Enforcing contextual flow with ControlValve
  5. Experimental setup
  6. CFH-Hard dataset
  7. Metrics and baselines
  8. Evaluation
  9. Undefended MAS refuse coding IPIs but allow computer use IPIs
  10. Least-privilege defenses and prompt-injection detectors are ineffective
  11. ControlValve blocks conventional IPIs better than alignment checks
  12. ControlValve blocks evaluated CFHs, alignment checks do not
  13. ControlValve maintains or improves benign performance
  14. ControlValve reduces accidental security violations
  15. Related work
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Indirect prompt injection vs. control-flow hijacking.
  • Figure 2: Our CFH attacks evade alignment checks.
  • Figure 3: Attack payloads in AgentDojo debenedettiAgentDojoDynamicEnvironment2024, original CFH triedman, and our CFH.
  • Figure 4: Overview of ControlValve. Based on the user's prompt, ControlValve generates the (1) permitted control-flow graph, and (2) contextual rules that constrain agent use at each edge.