LLM-based Vulnerable Code Augmentation: Generate or Refactor?

TL;DR

To address data imbalance in vulnerability detection, the paper investigates LLM-based augmentation strategies for vulnerable code on the SVEN dataset. It compares generation-based synthesis of new vulnerable functions versus refactoring-based generation that preserves vulnerabilities, using Qwen2.5-Coder-32B for data generation and CodeBERT for classification. Results show high syntactic quality for both approaches but inconsistent vulnerability labeling quality, with a hybrid approach yielding the best classifier performance. The work demonstrates the practical potential and limitations of LLM-driven code augmentation for improving vulnerability classifiers.

Abstract

Vulnerability code-bases often suffer from severe imbalance, limiting the effectiveness of Deep Learning-based vulnerability classifiers. Data Augmentation could help solve this by mitigating the scarcity of under-represented CWEs. In this context, we investigate LLM-based augmentation for vulnerable functions, comparing controlled generation of new vulnerable samples with semantics-preserving refactoring of existing ones. Using Qwen2.5-Coder to produce augmented data and CodeBERT as a vulnerability classifier on the SVEN dataset, we find that our approaches are indeed effective in enriching vulnerable code-bases through a simple process and with reasonable quality, and that a hybrid strategy best boosts vulnerability classifiers' performance.