Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vulnerability Detection in Popular Programming Languages with Language Models

Syafiq Al Atiiq, Christian Gehrmann, Kevin Dahlén

TL;DR

The paper addresses vulnerability detection in software using language models across multiple popular programming languages beyond C/C++. It curates and preprocesses the CVEFixes dataset to create language-specific subsets and evaluates four SOTA LM variants (CodeT5, CodeBERT, UnixCoder, DeepSeek-Coder) on JavaScript, Java, Python, PHP, Go, and C/C++. It provides cross-language comparisons, cross-dataset validations, and an analysis of the relationship between code complexity and detection performance, releasing the curated dataset and scripts for open science. The findings show language-dependent performance, with JavaScript and Java achieving relatively higher vulnerability-detection effectiveness than C/C++, while complexity metrics weakly correlate with performance, guiding future cross-language benchmarking and tool development.

Abstract

Vulnerability detection is crucial for maintaining software security, and recent research has explored the use of Language Models (LMs) for this task. While LMs have shown promising results, their performance has been inconsistent across datasets, particularly when generalizing to unseen code. Moreover, most studies have focused on the C/C++ programming language, with limited attention given to other popular languages. This paper addresses this gap by investigating the effectiveness of LMs for vulnerability detection in JavaScript, Java, Python, PHP, and Go, in addition to C/C++ for comparison. We utilize the CVEFixes dataset to create a diverse collection of language-specific vulnerabilities and preprocess the data to ensure quality and integrity. We fine-tune and evaluate state-of-the-art LMs across the selected languages and find that the performance of vulnerability detection varies significantly. JavaScript exhibits the best performance, with considerably better and more practical detection capabilities compared to C/C++. We also examine the relationship between code complexity and detection performance across the six languages and find only a weak correlation between code complexity metrics and the models' F1 scores.

Vulnerability Detection in Popular Programming Languages with Language Models

TL;DR

The paper addresses vulnerability detection in software using language models across multiple popular programming languages beyond C/C++. It curates and preprocesses the CVEFixes dataset to create language-specific subsets and evaluates four SOTA LM variants (CodeT5, CodeBERT, UnixCoder, DeepSeek-Coder) on JavaScript, Java, Python, PHP, Go, and C/C++. It provides cross-language comparisons, cross-dataset validations, and an analysis of the relationship between code complexity and detection performance, releasing the curated dataset and scripts for open science. The findings show language-dependent performance, with JavaScript and Java achieving relatively higher vulnerability-detection effectiveness than C/C++, while complexity metrics weakly correlate with performance, guiding future cross-language benchmarking and tool development.

Abstract

Vulnerability detection is crucial for maintaining software security, and recent research has explored the use of Language Models (LMs) for this task. While LMs have shown promising results, their performance has been inconsistent across datasets, particularly when generalizing to unseen code. Moreover, most studies have focused on the C/C++ programming language, with limited attention given to other popular languages. This paper addresses this gap by investigating the effectiveness of LMs for vulnerability detection in JavaScript, Java, Python, PHP, and Go, in addition to C/C++ for comparison. We utilize the CVEFixes dataset to create a diverse collection of language-specific vulnerabilities and preprocess the data to ensure quality and integrity. We fine-tune and evaluate state-of-the-art LMs across the selected languages and find that the performance of vulnerability detection varies significantly. JavaScript exhibits the best performance, with considerably better and more practical detection capabilities compared to C/C++. We also examine the relationship between code complexity and detection performance across the six languages and find only a weak correlation between code complexity metrics and the models' F1 scores.

Paper Structure

This paper contains 23 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Vulnerability Detection Techniques
  4. Traditional Approaches
  5. Non-Language Model-based Deep Learning Approaches
  6. Vulnerability Detection with Language Model
  7. LM for Code Analysis
  8. Vulnerability Detection using LM
  9. Problem Definition
  10. Dataset, Models and Setup
  11. Dataset
  12. Data Preprocessing and Filtering
  13. Models
  14. Experimental Setup
  15. Results and Analysis
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Distribution of programming languages in the CVEfixes dataset.
  • Figure 2: Mean Halstead effort vs. F1 score for each programming language.
  • Figure 3: Mean cyclomatic complexity vs. F1 score for each programming language.