Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoK: A Systems Perspective on Compound AI Threats and Countermeasures

Sarbartha Banerjee, Prateek Sahu, Mulong Luo, Anjo Vahldiek-Oberwagner, Neeraja J. Yadwadkar, Mohit Tiwari

TL;DR

This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack.

Abstract

Large language models (LLMs) used across enterprises often use proprietary models and operate on sensitive inputs and data. The wide range of attack vectors identified in prior research - targeting various software and hardware components used in training and inference - makes it extremely challenging to enforce confidentiality and integrity policies. As we advance towards constructing compound AI inference pipelines that integrate multiple large language models (LLMs), the attack surfaces expand significantly. Attackers now focus on the AI algorithms as well as the software and hardware components associated with these systems. While current research often examines these elements in isolation, we find that combining cross-layer attack observations can enable powerful end-to-end attacks with minimal assumptions about the threat model. Given, the sheer number of existing attacks at each layer, we need a holistic and systemized understanding of different attack vectors at each layer. This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack. Next, we systematize the ML attacks in lines with the Mitre Att&ck framework to better position each attack based on the threat model. Finally, we outline the existing countermeasures for both software and hardware layers and discuss the necessity of a comprehensive defense strategy to enable the secure and high-performance deployment of compound AI systems.

SoK: A Systems Perspective on Compound AI Threats and Countermeasures

TL;DR

This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack.

Abstract

Large language models (LLMs) used across enterprises often use proprietary models and operate on sensitive inputs and data. The wide range of attack vectors identified in prior research - targeting various software and hardware components used in training and inference - makes it extremely challenging to enforce confidentiality and integrity policies. As we advance towards constructing compound AI inference pipelines that integrate multiple large language models (LLMs), the attack surfaces expand significantly. Attackers now focus on the AI algorithms as well as the software and hardware components associated with these systems. While current research often examines these elements in isolation, we find that combining cross-layer attack observations can enable powerful end-to-end attacks with minimal assumptions about the threat model. Given, the sheer number of existing attacks at each layer, we need a holistic and systemized understanding of different attack vectors at each layer. This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack. Next, we systematize the ML attacks in lines with the Mitre Att&ck framework to better position each attack based on the threat model. Finally, we outline the existing countermeasures for both software and hardware layers and discuss the necessity of a comprehensive defense strategy to enable the secure and high-performance deployment of compound AI systems.

Paper Structure

This paper contains 24 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Threat Model Categorization
  4. MITRE ATT&CK Framework
  5. Threat Models
  6. Secret Assets
  7. Trust Entities
  8. Software vulnerabilities and defenses
  9. Attacks
  10. Defenses
  11. Key Takeaways
  12. Hardware vulnerabilities and defenses
  13. Attacks
  14. Defenses
  15. Key Takeaways
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Application, software and hardware layers for compound AI. Example shows cross-layer components can be exploited to leak data.
  • Figure 2: A Compound AI Pipeline begins with Query pre-processing to refine the input query and feed to an LLM agent. The agent extracts knowledge in the Retrieval stage, generates a draft response, and fills information generated from MoE experts in the Generation step. This response goes through compliance and fact checks Query post-processing step to finally generate a multi-modal output.
  • Figure 3: Positioning each attack widget in the Mitre Attack framework. The different colors specify the attacker capability ranging from remote access (weakest adversary) to physical access (strongest adversary). The attack steps through different steps. The impact denotes the security policy violation [C = Confidentiality, I = Integrity, A = Availability]
  • Figure 4: Three attack cases which leverages cross-stack vulnerabilities to mount powerful attacks. We map different attack techniques to the MITRE framework.