Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Preliminary Study of Large Language Models for Multilingual Vulnerability Detection

Junji Yu, Honglin Shu, Michael Fu, Dong Wang, Chakkrit Tantithamthavorn, Yasutaka Kamei, Junjie Chen

TL;DR

This work benchmarks PLMs and LLMs for multilingual vulnerability detection across seven programming languages using the REEF dataset, revealing that encoder-decoder PLMs, especially CodeT5P, deliver the strongest overall performance. While LLMs exhibit potential in certain languages and vulnerability types, they generally underperform compared with specialized PLMs in zero-shot vulnerability detection. The study also analyzes performance across languages and across the top 25 most dangerous CWE-IDs, showing closed-source LLMs often surpass open-source ones and highlighting the impact of model scale. Collectively, the results provide practical guidance for selecting language models in multilingual AVD and outline avenues to enhance LLM-based vulnerability detection via advanced prompting and integration of vulnerability knowledge.

Abstract

Deep learning-based approaches, particularly those leveraging pre-trained language models (PLMs), have shown promise in automated software vulnerability detection. However, existing methods are predominantly limited to specific programming languages, restricting their applicability in multilingual settings. Recent advancements in large language models (LLMs) offer language-agnostic capabilities and enhanced semantic understanding, presenting a potential solution to this limitation. While existing studies have explored LLMs for vulnerability detection, their detection performance remains unknown for multilingual vulnerabilities. To address this gap, we conducted a preliminary study to evaluate the effectiveness of PLMs and state-of-the-art LLMs across seven popular programming languages. Our findings reveal that the PLM CodeT5P achieves the best performance in multilingual vulnerability detection, particularly in identifying the most critical vulnerabilities. Based on these results, we further discuss the potential of LLMs in advancing real-world multilingual vulnerability detection. This work represents an initial step toward exploring PLMs and LLMs for cross-language vulnerability detection, offering key insights for future research and practical deployment.

A Preliminary Study of Large Language Models for Multilingual Vulnerability Detection

TL;DR

This work benchmarks PLMs and LLMs for multilingual vulnerability detection across seven programming languages using the REEF dataset, revealing that encoder-decoder PLMs, especially CodeT5P, deliver the strongest overall performance. While LLMs exhibit potential in certain languages and vulnerability types, they generally underperform compared with specialized PLMs in zero-shot vulnerability detection. The study also analyzes performance across languages and across the top 25 most dangerous CWE-IDs, showing closed-source LLMs often surpass open-source ones and highlighting the impact of model scale. Collectively, the results provide practical guidance for selecting language models in multilingual AVD and outline avenues to enhance LLM-based vulnerability detection via advanced prompting and integration of vulnerability knowledge.

Abstract

Deep learning-based approaches, particularly those leveraging pre-trained language models (PLMs), have shown promise in automated software vulnerability detection. However, existing methods are predominantly limited to specific programming languages, restricting their applicability in multilingual settings. Recent advancements in large language models (LLMs) offer language-agnostic capabilities and enhanced semantic understanding, presenting a potential solution to this limitation. While existing studies have explored LLMs for vulnerability detection, their detection performance remains unknown for multilingual vulnerabilities. To address this gap, we conducted a preliminary study to evaluate the effectiveness of PLMs and state-of-the-art LLMs across seven popular programming languages. Our findings reveal that the PLM CodeT5P achieves the best performance in multilingual vulnerability detection, particularly in identifying the most critical vulnerabilities. Based on these results, we further discuss the potential of LLMs in advancing real-world multilingual vulnerability detection. This work represents an initial step toward exploring PLMs and LLMs for cross-language vulnerability detection, offering key insights for future research and practical deployment.
Paper Structure (15 sections, 4 equations, 3 figures, 3 tables)

This paper contains 15 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Study Design
  4. Dataset Preparation
  5. Pre-trained Language Models
  6. Large Language Models
  7. Evaluation Metrics
  8. Implementation and Environment
  9. Results
  10. RQ1: How effective are PLMs and LLMs for multilingual vulnerability detection?
  11. RQ2: How do PLMs and LLMs perform across different programming language vulnerabilities?
  12. RQ3: How do PLMs and LLMs perform in detecting serious vulnerabilities?
  13. Future Work
  14. Threats to Validity
  15. Conclusion

Figures (3)

  • Figure 1: Template of zero-shot prompting strategy for automated vulnerability detection
  • Figure 2: Performance of PLM techniques on vulnerability detection across seven languages (y-axis: accuracy)
  • Figure 3: Performance of LLM techniques on vulnerability detection across seven languages (y-axis: accuracy)