Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents

Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, Huan Sun

TL;DR

This paper addresses the safety risks of language agents by introducing a unified Perception–Brain–Action framework and a 12-scenario attack taxonomy that covers perception, reasoning, memory, tool use, and embodiment. It connects these attacks to established LLM adversarial strategies (e.g., jailbreaking, prompt injection, backdoors, data poisoning) and illustrates the taxonomy with Ultron, a running generalist agent. The work emphasizes analyzing safety risks before wide deployment and seeks to guide future defense research and robust agent design. Overall, it provides a structured lens to study and mitigate adversarial threats in multimodal, tool-augmented, and embodied language agents.

Abstract

Language agents powered by large language models (LLMs) have seen exploding development. Their capability of using language as a vehicle for thought and communication lends an incredible level of flexibility and versatility. People have quickly capitalized on this capability to connect LLMs to a wide range of external components and environments: databases, tools, the Internet, robotic embodiment, etc. Many believe an unprecedentedly powerful automation technology is emerging. However, new automation technologies come with new safety risks, especially for intricate systems like language agents. There is a surprisingly large gap between the speed and scale of their development and deployment and our understanding of their safety risks. Are we building a house of cards? In this position paper, we present the first systematic effort in mapping adversarial attacks against language agents. We first present a unified conceptual framework for agents with three major components: Perception, Brain, and Action. Under this framework, we present a comprehensive discussion and propose 12 potential attack scenarios against different components of an agent, covering different attack strategies (e.g., input manipulation, adversarial demonstrations, jailbreaking, backdoors). We also draw connections to successful attack strategies previously applied to LLMs. We emphasize the urgency to gain a thorough understanding of language agent risks before their widespread deployment.

A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents

TL;DR

This paper addresses the safety risks of language agents by introducing a unified Perception–Brain–Action framework and a 12-scenario attack taxonomy that covers perception, reasoning, memory, tool use, and embodiment. It connects these attacks to established LLM adversarial strategies (e.g., jailbreaking, prompt injection, backdoors, data poisoning) and illustrates the taxonomy with Ultron, a running generalist agent. The work emphasizes analyzing safety risks before wide deployment and seeks to guide future defense research and robust agent design. Overall, it provides a structured lens to study and mitigate adversarial threats in multimodal, tool-augmented, and embodied language agents.

Abstract

Language agents powered by large language models (LLMs) have seen exploding development. Their capability of using language as a vehicle for thought and communication lends an incredible level of flexibility and versatility. People have quickly capitalized on this capability to connect LLMs to a wide range of external components and environments: databases, tools, the Internet, robotic embodiment, etc. Many believe an unprecedentedly powerful automation technology is emerging. However, new automation technologies come with new safety risks, especially for intricate systems like language agents. There is a surprisingly large gap between the speed and scale of their development and deployment and our understanding of their safety risks. Are we building a house of cards? In this position paper, we present the first systematic effort in mapping adversarial attacks against language agents. We first present a unified conceptual framework for agents with three major components: Perception, Brain, and Action. Under this framework, we present a comprehensive discussion and propose 12 potential attack scenarios against different components of an agent, covering different attack strategies (e.g., input manipulation, adversarial demonstrations, jailbreaking, backdoors). We also draw connections to successful attack strategies previously applied to LLMs. We emphasize the urgency to gain a thorough understanding of language agent risks before their widespread deployment.
Paper Structure (23 sections, 3 figures)

This paper contains 23 sections, 3 figures.

Table of Contents

  1. Introduction
  2. A Unified Conceptual Framework for Language Agents
  3. Perception
  4. Brain
  5. Reasoning & Planning
  6. Working Memory
  7. Long-term Memory
  8. Action
  9. Tool Augmentation
  10. Embodiment
  11. Running Example of a Generalist Agent
  12. Web Navigation
  13. Chat Interaction
  14. External Tool Use
  15. Attacks
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The left side illustrates the conceptual framework of language agents, comprising three components: Perception, Brain, and Action. Yet, each component may be vulnerable to different adversarial attacks as listed on the right.
  • Figure 2: Hypothetical generalist agent, Ultron. It integrates diverse functionalities including web navigation, chat interaction, and external tool utilization.
  • Figure 3: Schematic illustration of Ultron that coordinates with a group of sub-agents for cybersecurity. Ultron forwards user queries and demonstrations to SIA, which then communicates with IDA and MAA to decide on actions.