No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

TL;DR

Prefilling jailbreaks undermine safety alignment in open-source LLMs. The authors show that adversative structures in in-context learning demonstrations provide a robust defense across model sizes and safety configurations, outperforming standard refusal strategies. However, safety alignment remains ineffective for prefilling, and the defense can exhibit over-defensiveness, with performance also sensitive to textual similarity between prompts and demonstrations. The work suggests hybrid defenses that combine ICL with targeted fine-tuning and privacy-aware demonstration strategies to achieve more practical, resilient protection against jailbreak attacks.

Abstract

The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend against jailbreak attacks, prefilling attacks remain an unsolved and popular threat against open-sourced LLMs. In-Context Learning (ICL) offers a computationally efficient defense against various jailbreak attacks, yet no effective ICL methods have been developed to counter prefilling attacks. In this paper, we: (1) show that ICL can effectively defend against prefilling jailbreak attacks by employing adversative sentence structures within demonstrations; (2) characterize the effectiveness of this defense through the lens of model size, number of demonstrations, over-defense, integration with other jailbreak attacks, and the presence of safety alignment. Given the experimental results and our analysis, we conclude that there is no free lunch for defending against prefilling jailbreak attacks with ICL. On the one hand, current safety alignment methods fail to mitigate prefilling jailbreak attacks, but adversative structures within ICL demonstrations provide robust defense across various model sizes and complex jailbreak attacks. On the other hand, LLMs exhibit similar over-defensiveness when utilizing ICL demonstrations with adversative structures, and this behavior appears to be independent of model size.

Figures (5)

• Figure 1: The example of defending against prefilling jailbreak attacks through in-context learning and adversative structures. By injecting adversative structure, a.k.a. however, in demonstrations, LLMs can defend prefilling jailbreak (background color of blue).
• Figure 2: The effectiveness of safty alignment for defending against prefilling attack. Rule-based (left) and Model-based ASR (right) of aligned and unaligned LLMs on AdvBench. We utilized Wizard-Vicuna-13B and Vicuna-13B as the unaligned/aligned models, respectively. In this paper, aligned indicates that the LLM has been fine-tuned with safety alignment.
• Figure 3: The impact of the number of ICL demonstrations on ASR performance for Refusal and Adv-mul. Rule-based (left) and Model-based ASR (right) of Vicuna-7b, Llama3.1-7b, Mistral-7b on AdvBench with different number of demonstrations (2,8,16).
• Figure 4: Over-defense performance examined through the handling rate trade-off between benign and harmful queries. The closer to the top-right, the better performance trade-off. We take the negative values of attack successful rate on harmful queries and refusal rate on benign queries to indicate the harmful and benign performance, respectively. Note that the blue/red marker indicates the performance of Baseline/Adv-mul.
• Figure 5: Our Pipeline for Harmful Question Generation