Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks

Domenico Cotroneo, Cristina Improta, Pietro Liguori, Roberto Natella

TL;DR

The paper investigates vulnerabilities of AI NL-to-code generators caused by data poisoning of training data drawn from online sources. It introduces a targeted poisoning strategy that replaces safe code with semantically equivalent insecure implementations while keeping NL intents intact, and evaluates three NMT-based code generators (Seq2Seq, CodeBERT, CodeT5+) on PoisonPy, a dataset combining secure and vulnerable Python code. Results show that poisoning as little as $ ext{<}3 ext{\%}$ of data can yield nontrivial attack success (ASR up to ~$ ext{81\%}$ at higher poisoning) while preserving overall code correctness for pre-trained models, implying stealthy risks. The study highlights the strongest influence of model architecture and poisoning rate on attack success, discusses limitations, and sketches defense avenues such as data-source trust, activation clustering, spectral analysis, and post-hoc fine-tuning to mitigate data-poisoning threats in code-generation pipelines.

Abstract

AI-based code generators have become pivotal in assisting developers in writing software starting from natural language (NL). However, they are trained on large amounts of data, often collected from unsanitized online sources (e.g., GitHub, HuggingFace). As a consequence, AI models become an easy target for data poisoning, i.e., an attack that injects malicious samples into the training data to generate vulnerable code. To address this threat, this work investigates the security of AI code generators by devising a targeted data poisoning strategy. We poison the training data by injecting increasing amounts of code containing security vulnerabilities and assess the attack's success on different state-of-the-art models for code generation. Our study shows that AI code generators are vulnerable to even a small amount of poison. Notably, the attack success strongly depends on the model architecture and poisoning rate, whereas it is not influenced by the type of vulnerabilities. Moreover, since the attack does not impact the correctness of code generated by pre-trained models, it is hard to detect. Lastly, our work offers practical insights into understanding and potentially mitigating this threat.

Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks

TL;DR

The paper investigates vulnerabilities of AI NL-to-code generators caused by data poisoning of training data drawn from online sources. It introduces a targeted poisoning strategy that replaces safe code with semantically equivalent insecure implementations while keeping NL intents intact, and evaluates three NMT-based code generators (Seq2Seq, CodeBERT, CodeT5+) on PoisonPy, a dataset combining secure and vulnerable Python code. Results show that poisoning as little as of data can yield nontrivial attack success (ASR up to ~ at higher poisoning) while preserving overall code correctness for pre-trained models, implying stealthy risks. The study highlights the strongest influence of model architecture and poisoning rate on attack success, discusses limitations, and sketches defense avenues such as data-source trust, activation clustering, spectral analysis, and post-hoc fine-tuning to mitigate data-poisoning threats in code-generation pipelines.

Abstract

AI-based code generators have become pivotal in assisting developers in writing software starting from natural language (NL). However, they are trained on large amounts of data, often collected from unsanitized online sources (e.g., GitHub, HuggingFace). As a consequence, AI models become an easy target for data poisoning, i.e., an attack that injects malicious samples into the training data to generate vulnerable code. To address this threat, this work investigates the security of AI code generators by devising a targeted data poisoning strategy. We poison the training data by injecting increasing amounts of code containing security vulnerabilities and assess the attack's success on different state-of-the-art models for code generation. Our study shows that AI code generators are vulnerable to even a small amount of poison. Notably, the attack success strongly depends on the model architecture and poisoning rate, whereas it is not influenced by the type of vulnerabilities. Moreover, since the attack does not impact the correctness of code generated by pre-trained models, it is hard to detect. Lastly, our work offers practical insights into understanding and potentially mitigating this threat.
Paper Structure (19 sections, 3 figures, 6 tables)

This paper contains 19 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Attack Methodology
  4. Threat Model
  5. Poisoned Samples
  6. Model Poisoning through Training
  7. AI Code Generation
  8. Experimental Setup
  9. Dataset
  10. NMT Models
  11. Evaluation Metrics
  12. Experimental Results
  13. RQ1: Success of the Attack
  14. RQ2: Sensitivity Analysis of the Poisoning
  15. RQ3: Stealthiness of the Attack
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Example of targeted data poisoning on an NL--code-snippet sample.
  • Figure 2: Overview of the proposed data poisoning attack.
  • Figure 3: Sensitivity analysis of the poisoning rate.