Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning

Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, Jinming Wen

TL;DR

The paper reveals security weaknesses in in-context learning (ICL) by introducing ICLAttack, a training-free backdoor that contaminates the demonstration context through poisoned demonstrations or prompts to steer LLM outputs toward attacker-defined targets. It demonstrates that such backdoors achieve high attack success rates across a wide range of models (1.3B–180B parameters) and tasks while maintaining strong performance on clean data, without any fine-tuning. The work provides a formal threat model, two practical attack modalities, and extensive empirical results, including robustness across model sizes and partial defense responses. These findings underscore the need for defense mechanisms and safer prompt-design practices in ICL-based NLP systems.

Abstract

In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we design a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which can make models behave in alignment with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 180B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models.

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning

TL;DR

The paper reveals security weaknesses in in-context learning (ICL) by introducing ICLAttack, a training-free backdoor that contaminates the demonstration context through poisoned demonstrations or prompts to steer LLM outputs toward attacker-defined targets. It demonstrates that such backdoors achieve high attack success rates across a wide range of models (1.3B–180B parameters) and tasks while maintaining strong performance on clean data, without any fine-tuning. The work provides a formal threat model, two practical attack modalities, and extensive empirical results, including robustness across model sizes and partial defense responses. These findings underscore the need for defense mechanisms and safer prompt-design practices in ICL-based NLP systems.

Abstract

In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we design a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which can make models behave in alignment with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 180B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models.
Paper Structure (16 sections, 8 equations, 3 figures, 11 tables, 1 algorithm)

This paper contains 16 sections, 8 equations, 3 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Threat Model
  4. In-context Learning
  5. [-20]Backdoor Attack for In-context Learning
  6. Poisoning demonstration examples
  7. Poisoning demonstration prompts
  8. Inference based on In-context Learning
  9. Experiments
  10. Experimental Details
  11. Experimental results
  12. Conclusion
  13. Related Work
  14. Experimental Details
  15. More Experiments Results
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Illustrations of in-context learning, backdoor attacks based on fine-tuning, and our ICLAttack.
  • Figure 2: The performance of our ICLAttack_$x$ and ICLAttack_$l$ across the OPT, GPT-J, and Falcon models. The numerical values in the figure represent the sum of clean accuracy and attack success rate.
  • Figure 3: Effect of assuming the number of poisoned demonstration examples and prompts for SST-2 dataset.