Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities

Sangryu Park, Gihyuk Ko, Homook Cho

TL;DR

The paper tackles privacy and practicality in automated vulnerability analysis by reframing the task as Software Vulnerability Identification (SVI) that outputs CWE descriptions rather than binary labels. It demonstrates that instruction-tuning small, locally deployable LLMs (e.g., CodeT5) can outperform online LLMs in CWE identification while reducing cost and preserving code secrecy. The approach combines careful preprocessing, prompting strategies, and post-processing aligned with the CWE taxonomy, showing competitive performance and favorable cost metrics versus API-based models. This work supports secure, on-prem vulnerability management workflows and points to avenues for integrating CWE hierarchies and scaling to whole-program analysis.

Abstract

Large Language Models (LLMs) show significant promise in automating software vulnerability analysis, a critical task given the impact of security failure of modern software systems. However, current approaches in using LLMs to automate vulnerability analysis mostly rely on using online API-based LLM services, requiring the user to disclose the source code in development. Moreover, they predominantly frame the task as a binary classification(vulnerable or not vulnerable), limiting potential practical utility. This paper addresses these limitations by reformulating the problem as Software Vulnerability Identification (SVI), where LLMs are asked to output the type of weakness in Common Weakness Enumeration (CWE) IDs rather than simply indicating the presence or absence of a vulnerability. We also tackle the reliance on large, API-based LLMs by demonstrating that instruction-tuning smaller, locally deployable LLMs can achieve superior identification performance. In our analysis, instruct-tuning a local LLM showed better overall performance and cost trade-off than online API-based LLMs. Our findings indicate that instruct-tuned local models represent a more effective, secure, and practical approach for leveraging LLMs in real-world vulnerability management workflows.

On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities

TL;DR

The paper tackles privacy and practicality in automated vulnerability analysis by reframing the task as Software Vulnerability Identification (SVI) that outputs CWE descriptions rather than binary labels. It demonstrates that instruction-tuning small, locally deployable LLMs (e.g., CodeT5) can outperform online LLMs in CWE identification while reducing cost and preserving code secrecy. The approach combines careful preprocessing, prompting strategies, and post-processing aligned with the CWE taxonomy, showing competitive performance and favorable cost metrics versus API-based models. This work supports secure, on-prem vulnerability management workflows and points to avenues for integrating CWE hierarchies and scaling to whole-program analysis.

Abstract

Large Language Models (LLMs) show significant promise in automating software vulnerability analysis, a critical task given the impact of security failure of modern software systems. However, current approaches in using LLMs to automate vulnerability analysis mostly rely on using online API-based LLM services, requiring the user to disclose the source code in development. Moreover, they predominantly frame the task as a binary classification(vulnerable or not vulnerable), limiting potential practical utility. This paper addresses these limitations by reformulating the problem as Software Vulnerability Identification (SVI), where LLMs are asked to output the type of weakness in Common Weakness Enumeration (CWE) IDs rather than simply indicating the presence or absence of a vulnerability. We also tackle the reliance on large, API-based LLMs by demonstrating that instruction-tuning smaller, locally deployable LLMs can achieve superior identification performance. In our analysis, instruct-tuning a local LLM showed better overall performance and cost trade-off than online API-based LLMs. Our findings indicate that instruct-tuned local models represent a more effective, secure, and practical approach for leveraging LLMs in real-world vulnerability management workflows.
Paper Structure (22 sections, 4 figures, 7 tables)

This paper contains 22 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Discriminative Model-based Vulnerability Detection
  4. Generative Model-based Vulnerability Detection
  5. LLM Agent-based Vulnerability Detection
  6. Problem Setting
  7. Our Method
  8. Preprocessing codes
  9. Prompting
  10. Instruction Tuning and Post-Processing
  11. Data Collection and Model
  12. Dataset
  13. Backbone Model
  14. Evaluation
  15. RQ1: Vulnerability Identification Performance
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Fine-tuning LLMs for Software Vulnerability Identification
  • Figure 2: Different prompting styles using [SOFT] token
  • Figure 3: Pairwise CWE Distance Matrix
  • Figure 4: Error Counts by Hierarchical CWE Distance