Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM

TL;DR

The paper tackles vulnerability fix detection in open-source software by identifying limitations of code-only approaches and proposing LLM4VFD, a framework that fuses Code Change Intention, Development Artifacts, and Historical Vulnerability signals with Chain-of-Thought reasoning and In-Context Learning. It introduces three components (CCI, DA, HV) feeding into CAVFD, enabling not only a prediction but also an explanatory analysis to assist security experts. Experiments on the BigVulFixes dataset show LLM4VFD substantially outperforms state-of-the-art PLM-based methods, with notable gains on smaller models and strong ablation results confirming each component’s value. A user study indicates the generated analyses improve experts’ efficiency in identifying vulnerability fixes, underscoring practical impact and yielding a publicly released dataset for reproducibility.

Abstract

Detecting vulnerability fix commits in open-source software is crucial for maintaining software security. To help OSS identify vulnerability fix commits, several automated approaches are developed. However, existing approaches like VulFixMiner and CoLeFunDa, focus solely on code changes, neglecting essential context from development artifacts. Tools like Vulcurator, which integrates issue reports, fail to leverage semantic associations between different development artifacts (e.g., pull requests and history vulnerability fixes). Moreover, they miss vulnerability fixes in tangled commits and lack explanations, limiting practical use. Hence to address those limitations, we propose LLM4VFD, a novel framework that leverages Large Language Models (LLMs) enhanced with Chain-of-Thought reasoning and In-Context Learning to improve the accuracy of vulnerability fix detection. LLM4VFD comprises three components: (1) Code Change Intention, which analyzes commit summaries, purposes, and implications using Chain-of-Thought reasoning; (2) Development Artifact, which incorporates context from related issue reports and pull requests; (3) Historical Vulnerability, which retrieves similar past vulnerability fixes to enrich context. More importantly, on top of the prediction, LLM4VFD also provides a detailed analysis and explanation to help security experts understand the rationale behind the decision. We evaluated LLM4VFD against state-of-the-art techniques, including Pre-trained Language Model-based approaches and vanilla LLMs, using a newly collected dataset, BigVulFixes. Experimental results demonstrate that LLM4VFD significantly outperforms the best-performed existing approach by 68.1%--145.4%. Furthermore, We conducted a user study with security experts, showing that the analysis generated by LLM4VFD improves the efficiency of vulnerability fix identification.

Figures (7)

• Figure 1: An example of tangled commit with 164 lines changed, while only two lines (in red box) are related to vulnerability fix. github.spring.202fa5c
• Figure 2: An example of a commit github.gpac.c88df2e with only 2 lines changed (in red box), while related issues reports github.gpac.issues.2475 (in blue box) provided critical information.
• Figure 3: A commit only with small change by adding condition check (left github.protobufjs.e66379f) and its relevant historical vulnerability fix commit (right github.protobufjs.3357ef7).
• Figure 4: The framework of LLM4VFD.
• Figure 5: The prompt template of Code Change Intention.