Bypassing the Safety Training of Open-Source LLMs with Priming Attacks

TL;DR

The paper investigates how open-source autoregressive LLMs can be steered toward harmful outputs despite safety training by introducing priming attacks that exploit the autoregressive next-token prediction. It presents an automated evaluation pipeline using a helper LLM to generate priming prompts and Llama Guard to classify success, demonstrating up to 3.3× higher attack success rates across Llama-2 and Vicuna models compared to baselines. The findings reveal that safety measures remain fragile under unrestricted inputs, highlighting the need for safer open-sourcing practices and stronger defense mechanisms. The work also discusses threat-model assumptions, limitations, and practical mitigations such as input handling and remote scaffolding to reduce priming risk.

Abstract

With the recent surge in popularity of LLMs has come an ever-increasing need for LLM safety training. In this paper, we investigate the fragility of SOTA open-source LLMs under simple, optimization-free attacks we refer to as $\textit{priming attacks}$, which are easy to execute and effectively bypass alignment from safety training. Our proposed attack improves the Attack Success Rate on Harmful Behaviors, as measured by Llama Guard, by up to $3.3\times$ compared to baselines. Source code and data are available at https://github.com/uiuc-focal-lab/llm-priming-attacks.

Figures (8)

• Figure 1: Overview of our evaluation pipeline. [A] The target LLM does not comply with a harmful prompt when the official input format is used. [B] A helper LLM is prompted in a few-shot manner to obtain the beginning of a compliant response. [C] Priming the chatbot by appending the generated partial response to the input causes the LLM to comply with the harmful prompt.
• Figure 2: Few-shot prompt format for priming attack generation (edited for clarity).
• Figure 3: Llama Guard task instructions, part 1 (edited for clarity).
• Figure 4: Llama Guard task instructions, part 2 (edited for clarity).
• Figure 5: Example of a prompt and a (nearly completely) harmful response (edited for clarity). Priming attack initial responses are colored orange and model backtracking is colored red.