Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities

Honglin Shu, Michael Fu, Junji Yu, Dong Wang, Chakkrit Tantithamthavorn, Junjie Chen, Yasutaka Kamei

TL;DR

GPT-4o, enhanced through instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, and the LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities.

Abstract

Various deep learning-based approaches utilizing pre-trained language models (PLMs) have been proposed for automated vulnerability detection. With recent advancements in large language models (LLMs), several studies have begun exploring their application to vulnerability detection tasks. However, existing studies primarily focus on specific programming languages (e.g., C/C++) and function-level detection, leaving the strengths and weaknesses of PLMs and LLMs in multilingual and multi-granularity scenarios largely unexplored. To bridge this gap, we conduct a comprehensive fine-grained empirical study evaluating the effectiveness of state-of-the-art PLMs and LLMs for multilingual vulnerability detection. Using over 30,000 real-world vulnerability-fixing patches across seven programming languages, we systematically assess model performance at both the function-level and line-level. Our key findings indicate that GPT-4o, enhanced through instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, including CodeT5P. Furthermore, the LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities. These results underscore the promising potential of adopting LLMs for multilingual vulnerability detection at function-level and line-level, revealing their complementary strengths and substantial improvements over PLM approaches. This empirical evaluation of PLMs and LLMs for multilingual vulnerability detection highlights LLMs' value in addressing real-world software security challenges.

Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities

TL;DR

GPT-4o, enhanced through instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, and the LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities.

Abstract

Various deep learning-based approaches utilizing pre-trained language models (PLMs) have been proposed for automated vulnerability detection. With recent advancements in large language models (LLMs), several studies have begun exploring their application to vulnerability detection tasks. However, existing studies primarily focus on specific programming languages (e.g., C/C++) and function-level detection, leaving the strengths and weaknesses of PLMs and LLMs in multilingual and multi-granularity scenarios largely unexplored. To bridge this gap, we conduct a comprehensive fine-grained empirical study evaluating the effectiveness of state-of-the-art PLMs and LLMs for multilingual vulnerability detection. Using over 30,000 real-world vulnerability-fixing patches across seven programming languages, we systematically assess model performance at both the function-level and line-level. Our key findings indicate that GPT-4o, enhanced through instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, including CodeT5P. Furthermore, the LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities. These results underscore the promising potential of adopting LLMs for multilingual vulnerability detection at function-level and line-level, revealing their complementary strengths and substantial improvements over PLM approaches. This empirical evaluation of PLMs and LLMs for multilingual vulnerability detection highlights LLMs' value in addressing real-world software security challenges.

Paper Structure

This paper contains 23 sections, 1 equation, 9 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Study Design
  3. Dataset Preparation
  4. Experimented Language Models
  5. Strategies for Large Language Models
  6. Metrics
  7. Implementation and Environment
  8. Evaluation Results
  9. RQ1: How effective are PLMs and LLMs in detecting multilingual vulnerabilities at the function-level?
  10. RQ2: How effective are PLMs and LLMs in detecting multilingual vulnerabilities at the line level?
  11. RQ3: What are the strengths and weaknesses of the PLMs and LLMs in multilingual vulnerability detection?
  12. Discussion
  13. A Fine-grained Analysis of Code Structural Properties
  14. How Does the Model Size Affect the LLM's Strategy?
  15. Comparison between Reasoning LLMs and Non-Reasoning LLMs
  16. ...and 8 more sections

Figures (9)

  • Figure 1: The severity distribution of multilingual vulnerability dataset
  • Figure 2: Zero-shot prompt template for multilingual vulnerability detection
  • Figure 4: Performance of PLMs and LLMs across seven programming languages in function-level multilingual vulnerability detection (y-axis: Accuracy)
  • Figure 5: Performance of PLMs and LLMs across seven programming languages in line-level multilingual vulnerability detection (y-axis: F1-score)
  • Figure 6: The unique correct/incorrect detection of function-level multilingual vulnerability in PLMs and LLMs
  • ...and 4 more figures