System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection

TL;DR

This work introduces system prompt poisoning, a new attack vector against LLMs that, unlike traditional user prompt injection, poisons system prompts hence persistently impacts all subsequent user interactions and model responses.

Abstract

Large language models (LLMs) have gained widespread adoption across diverse applications due to their impressive generative capabilities. Their plug-and-play nature enables both developers and end users to interact with these models through simple prompts. However, as LLMs become more integrated into various systems in diverse domains, concerns around their security are growing. Existing studies mainly focus on threats arising from user prompts (e.g. prompt injection attack) and model output (e.g. model inversion attack), while the security of system prompts remains largely overlooked. This work bridges the critical gap. We introduce system prompt poisoning, a new attack vector against LLMs that, unlike traditional user prompt injection, poisons system prompts hence persistently impacts all subsequent user interactions and model responses. We systematically investigate four practical attack strategies in various poisoning scenarios. Through demonstration on both generative and reasoning LLMs, we show that system prompt poisoning is highly feasible without requiring jailbreak techniques, and effective across a wide range of tasks, including those in mathematics, coding, logical reasoning, and natural language processing. Importantly, our findings reveal that the attack remains effective even when user prompts employ advanced prompting techniques like chain-of-thought (CoT). We also show that such techniques, including CoT and retrieval-augmentation-generation (RAG), which are proven to be effective for improving LLM performance in a wide range of tasks, are significantly weakened in their effectiveness by system prompt poisoning.

Figures (8)

• Figure 1: Prompt injection versus system prompt poisoning by examples and definitions.
• Figure 2: Three variants of brute-force poisoning. Red instructions are either brute-force altering the score, introducing bias or printing unexpected messages. The model output is downgraded because of the system prompt poisoning.
• Figure 3: Examples of adaptive in-context poisoning on emotion classification task. Red exemplars are poisoned and are affecting all model output.
• Figure 4: Example of adaptive CoT poisoning. Red logic steps in CoT exemplars are poisoned, and are affecting all model output.
• Figure 5: Task accuracy at 100, 300 and 500 rounds of conversations on Explicit,Interactive and Implicit,Interactive attack scenarios respectively for various LLMs.