Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs in Code Vulnerability Analysis: A Proof of Concept

Shaznin Sultana, Sadia Afreen, Nasir U. Eisty

TL;DR

This work investigates open-source LLMs as integrated assistants for a four-stage software vulnerability workflow in C/C++: detection, severity and access complexity prediction, and fix generation. By comparing five code-specific paired with general-purpose models across prompting regimes (zero-shot, few-shot) and fine-tuning (LoRA PEFT), the study shows that fine-tuning consistently enhances performance, while code-specialized models do not uniformly outrank general models. It also demonstrates significant limitations in current repair-quality metrics, underscoring the need for execution-based evaluation. The findings offer a practical baseline for deploying LLM-assisted vulnerability analysis and remediation, with implications for model choice, prompting design, and evaluation strategies in real-world security pipelines.

Abstract

Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring intelligent automation to detect, assess, and remediate vulnerabilities more efficiently and accurately. Objective: This paper explores the incorporation of code-specific and general-purpose Large Language Models (LLMs) to automate critical software security tasks, such as identifying vulnerabilities, predicting severity and access complexity, and generating fixes as a proof of concept. Method: We evaluate five pairs of recent LLMs, including both code-based and general-purpose open-source models, on two recognized C/C++ vulnerability datasets, namely Big-Vul and Vul-Repair. Additionally, we compare fine-tuning and prompt-based approaches. Results: The results show that fine-tuning uniformly outperforms both zero-shot and few-shot approaches across all tasks and models. Notably, code-specialized models excel in zero-shot and few-shot settings on complex tasks, while general-purpose models remain nearly as effective. Discrepancies among CodeBLEU, CodeBERTScore, BLEU, and ChrF highlight the inadequacy of current metrics for measuring repair quality. Conclusions: This study contributes to the software security community by investigating the potential of advanced LLMs to improve vulnerability analysis and remediation.

LLMs in Code Vulnerability Analysis: A Proof of Concept

TL;DR

This work investigates open-source LLMs as integrated assistants for a four-stage software vulnerability workflow in C/C++: detection, severity and access complexity prediction, and fix generation. By comparing five code-specific paired with general-purpose models across prompting regimes (zero-shot, few-shot) and fine-tuning (LoRA PEFT), the study shows that fine-tuning consistently enhances performance, while code-specialized models do not uniformly outrank general models. It also demonstrates significant limitations in current repair-quality metrics, underscoring the need for execution-based evaluation. The findings offer a practical baseline for deploying LLM-assisted vulnerability analysis and remediation, with implications for model choice, prompting design, and evaluation strategies in real-world security pipelines.

Abstract

Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring intelligent automation to detect, assess, and remediate vulnerabilities more efficiently and accurately. Objective: This paper explores the incorporation of code-specific and general-purpose Large Language Models (LLMs) to automate critical software security tasks, such as identifying vulnerabilities, predicting severity and access complexity, and generating fixes as a proof of concept. Method: We evaluate five pairs of recent LLMs, including both code-based and general-purpose open-source models, on two recognized C/C++ vulnerability datasets, namely Big-Vul and Vul-Repair. Additionally, we compare fine-tuning and prompt-based approaches. Results: The results show that fine-tuning uniformly outperforms both zero-shot and few-shot approaches across all tasks and models. Notably, code-specialized models excel in zero-shot and few-shot settings on complex tasks, while general-purpose models remain nearly as effective. Discrepancies among CodeBLEU, CodeBERTScore, BLEU, and ChrF highlight the inadequacy of current metrics for measuring repair quality. Conclusions: This study contributes to the software security community by investigating the potential of advanced LLMs to improve vulnerability analysis and remediation.
Paper Structure (28 sections, 4 figures, 8 tables)

This paper contains 28 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. LLMs Selection
  5. Tasks Selection
  6. Detection
  7. Severity Prediction
  8. Access Complexity Classification
  9. Fix generation
  10. Dataset Exploration
  11. Hardware Setup
  12. Model Loading and Tokenization
  13. Prompt Engineering
  14. Zero-shot
  15. Few-Shot
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Prompt Design
  • Figure 2: Model Performance Comparison based on Zero-Shot and Few-Shot approach for task 1
  • Figure 3: Model Performance Comparison based on Zero-Shot and Few-Shot approach for task 2
  • Figure 4: Model Performance Comparison based on Zero-Shot and Few-Shot approach for task 3