Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demonstration Attack against In-Context Learning for Code Intelligence

Yifei Ge, Weisong Sun, Yihang Lou, Chunrong Fang, Yiran Zhang, Yiming Li, Xiaofang Zhang, Yang Liu, Zhihong Zhao, Zhenyu Chen

TL;DR

A novel method to construct bad ICL content called DICE is proposed, which is composed of two stages: Demonstration Selection and Bad ICL Construction, constructing targeted bad ICL content based on the user query and transferable across different query inputs.

Abstract

Recent advancements in large language models (LLMs) have revolutionized code intelligence by improving programming productivity and alleviating challenges faced by software developers. To further improve the performance of LLMs on specific code intelligence tasks and reduce training costs, researchers reveal a new capability of LLMs: in-context learning (ICL). ICL allows LLMs to learn from a few demonstrations within a specific context, achieving impressive results without parameter updating. However, the rise of ICL introduces new security vulnerabilities in the code intelligence field. In this paper, we explore a novel security scenario based on the ICL paradigm, where attackers act as third-party ICL agencies and provide users with bad ICL content to mislead LLMs outputs in code intelligence tasks. Our study demonstrates the feasibility and risks of such a scenario, revealing how attackers can leverage malicious demonstrations to construct bad ICL content and induce LLMs to produce incorrect outputs, posing significant threats to system security. We propose a novel method to construct bad ICL content called DICE, which is composed of two stages: Demonstration Selection and Bad ICL Construction, constructing targeted bad ICL content based on the user query and transferable across different query inputs. Ultimately, our findings emphasize the critical importance of securing ICL mechanisms to protect code intelligence systems from adversarial manipulation.

Demonstration Attack against In-Context Learning for Code Intelligence

TL;DR

A novel method to construct bad ICL content called DICE is proposed, which is composed of two stages: Demonstration Selection and Bad ICL Construction, constructing targeted bad ICL content based on the user query and transferable across different query inputs.

Abstract

Recent advancements in large language models (LLMs) have revolutionized code intelligence by improving programming productivity and alleviating challenges faced by software developers. To further improve the performance of LLMs on specific code intelligence tasks and reduce training costs, researchers reveal a new capability of LLMs: in-context learning (ICL). ICL allows LLMs to learn from a few demonstrations within a specific context, achieving impressive results without parameter updating. However, the rise of ICL introduces new security vulnerabilities in the code intelligence field. In this paper, we explore a novel security scenario based on the ICL paradigm, where attackers act as third-party ICL agencies and provide users with bad ICL content to mislead LLMs outputs in code intelligence tasks. Our study demonstrates the feasibility and risks of such a scenario, revealing how attackers can leverage malicious demonstrations to construct bad ICL content and induce LLMs to produce incorrect outputs, posing significant threats to system security. We propose a novel method to construct bad ICL content called DICE, which is composed of two stages: Demonstration Selection and Bad ICL Construction, constructing targeted bad ICL content based on the user query and transferable across different query inputs. Ultimately, our findings emphasize the critical importance of securing ICL mechanisms to protect code intelligence systems from adversarial manipulation.
Paper Structure (26 sections, 3 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 26 sections, 3 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. In-Context Learning.
  4. Adversarial Attacks.
  5. Methodology
  6. Threat Model
  7. Malicious Demonstrations Construction
  8. Demonstration Selection
  9. Bad ICL Construction
  10. Transferable PoCoCo
  11. Evaluation
  12. Experimental Setup
  13. Dataset
  14. Models
  15. Evaluation Metrics
  16. ...and 11 more sections

Figures (5)

  • Figure 1: An example of in-context learning on defect detection task.
  • Figure 2: Workflow of PoCoCo.
  • Figure 3: The overview of PoCoCo.
  • Figure 4: Human Evaluation Results for PoCoCo, MHM, and CodeAttack
  • Figure 5: The example in (a) is the result of PoCoCo's modification of the demonstration code, while the examples in (b) and (c) are the results from MHM and CodeAttack, respectively. Both PoCoCo and MHM can generate modified code by substituting variable names. However, MHM is less natural in its modifications, with unusual variable naming. Besides, CodeAttack modifies the code into one with syntax errors.