A Survey of Attacks on Large Language Models

TL;DR

This paper surveys adversarial threats to large language models and LLM-based agents, organizing attacks along Training, Inference, and Availability/Integrity phases. It presents a comprehensive taxonomy of backdoor, poisoning, jailbreaking, prompt injection, DoS, and watermarking attacks, including many agent-specific variants. The authors highlight the expanding attack surface with LLM-based agents and external tools, and critique existing defenses as often narrow or phase-specific. They call for unified threat benchmarks, cross-phase defense strategies, and adaptive, explainable defenses to improve robustness in real-world, multi-agent settings.

Abstract

Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

Figures (14)

• Figure 1: Example of next-token prediction. The raw text $\mathbf{X}$ is first tokenized into $\{x_1, x_2, \dots, x_n\}$ and mapped into input token vectors $\{V_{x_1}, V_{x_2}, \dots, V_{x_n}\}$ as the input to LLM $F(\cdot)$. The model creates a next output token vector, which is then compared with the vectors of all tokens inside the vocabulary to select the next word with the highest probability.
• Figure 2: Example of prompt and response operation on GPT-4.
• Figure 3: A taxonomy of attacks of LLMs and LLM-based agent systems. Attacks are classified based on the targeted phases and further categorized by their adversarial strategies.
• Figure 4: Example of backdoor attack on LLM-based sentiment analysis zhaosurvey. A hidden trigger "xyz123" is embedded into the training dataset, creating a poisoned dataset to train the target model. Under normal conditions, the model classifies sentiment correctly. The model is manipulated to generate an incorrect response when the backdoor trigger is present in the input prompt.
• Figure 5: Example of Homograph Replacement-based Attack li2021hidden. Selected characters in raw sentences are substituted with visually similar Unicode homographs, where the tokens of these characters are mapped into special characters such as "[UNK]".