Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finetuning Large Language Models for Vulnerability Detection

Alexey Shestov, Rodion Levichev, Ravil Mussabayev, Evgeny Maslov, Anton Cheshkov, Pavel Zadorozhny

TL;DR

This work investigates finetuning a large code-focused LLM, WizardCoder, for vulnerability detection in Java functions, treating the task as binary classification. It develops an efficient training workflow, including low-memory fine-tuning with LoRA, a classification-focused objective $L_{\text{class}}$, and a batch-packing strategy that speeds training by reducing padding. The study shows WizardCoder achieving state-of-the-art나 performance over CodeBERT-based baselines on balanced and imbalanced datasets, with modest gains from focal loss and sample weighting for imbalanced data. Overall, the results demonstrate the viability of transfer learning with large pretrained language models for specialized code analysis tasks, while highlighting avenues for further improvement such as curriculum learning and data augmentation.

Abstract

This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.

Finetuning Large Language Models for Vulnerability Detection

TL;DR

This work investigates finetuning a large code-focused LLM, WizardCoder, for vulnerability detection in Java functions, treating the task as binary classification. It develops an efficient training workflow, including low-memory fine-tuning with LoRA, a classification-focused objective , and a batch-packing strategy that speeds training by reducing padding. The study shows WizardCoder achieving state-of-the-art나 performance over CodeBERT-based baselines on balanced and imbalanced datasets, with modest gains from focal loss and sample weighting for imbalanced data. Overall, the results demonstrate the viability of transfer learning with large pretrained language models for specialized code analysis tasks, while highlighting avenues for further improvement such as curriculum learning and data augmentation.

Abstract

This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, and adapt it for vulnerability detection through further finetuning. To accelerate training, we modify WizardCoder's training procedure, also we investigate optimal training regimes. For the imbalanced dataset with many more negative examples than positive, we also explore different techniques to improve classification performance. The finetuned WizardCoder model achieves improvement in ROC AUC and F1 measures on balanced and imbalanced vulnerability datasets over CodeBERT-like model, demonstrating the effectiveness of adapting pretrained LLMs for vulnerability detection in source code. The key contributions are finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without the performance harm, optimizing the training procedure and regimes, handling class imbalance, and improving performance on difficult vulnerability detection datasets. This demonstrates the potential for transfer learning by finetuning large pretrained language models for specialized source code analysis tasks.
Paper Structure (39 sections, 2 equations, 2 figures, 10 tables)

This paper contains 39 sections, 2 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Implementation details
  4. Challenges
  5. LLM model selection for finetuning
  6. Adapting LLMs for limited computational resources
  7. Adapting LLMs for the classification task
  8. Improving a slow training speed
  9. Choosing an LLM to finetune
  10. Choosing the best StarCoder-family model
  11. Adapting LLMs for limited computational resources
  12. Adapting LLMs for classification
  13. Speeding up the classification training
  14. Experimental setup
  15. Datasets
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Distribution of the top 19 CWE categories and others in the test set with $P_3$.
  • Figure 2: Distribution of the top 19 CWE categories and others in the test set without $P_3$.