Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities

TL;DR

This work investigates chain-of-thought prompting for software vulnerability analysis by introducing Vulnerability-Semantics-guided Prompting (VSP), a unified prompting strategy that concentrates LLM reasoning on vulnerability semantics and their essential data/control-flow contexts. It systematically evaluates VSP on three core tasks—vulnerability identification, vulnerability discovery, and vulnerability patching—across two datasets (synthetic SARD and real-world CVE) and three LLMs (GPT-3.5, Llama2, Falcon). Across tasks, VSP consistently outperforms standard prompting and several baselines, achieving substantial gains in F1 for identification and discovery and high patching accuracy on synthetic data, with more challenging results on CVE data that reveal key failure modes. The authors also analyze failure causes (insufficient context, CWE oblivion, incomplete CF/DF analysis) and propose practical remedies, including augmenting context with code comments and integrating traditional static analysis, suggesting VSP as a promising direction for leveraging LLMs in defensive software vulnerability analysis and zero-day discovery.

Abstract

Security vulnerabilities are increasingly prevalent in modern software and they are widely consequential to our society. Various approaches to defending against these vulnerabilities have been proposed, among which those leveraging deep learning (DL) avoid major barriers with other techniques hence attracting more attention in recent years. However, DL-based approaches face critical challenges including the lack of sizable and quality-labeled task-specific datasets and their inability to generalize well to unseen, real-world scenarios. Lately, large language models (LLMs) have demonstrated impressive potential in various domains by overcoming those challenges, especially through chain-of-thought (CoT) prompting. In this paper, we explore how to leverage LLMs and CoT to address three key software vulnerability analysis tasks: identifying a given type of vulnerabilities, discovering vulnerabilities of any type, and patching detected vulnerabilities. We instantiate the general CoT methodology in the context of these tasks through VSP , our unified, vulnerability-semantics-guided prompting approach, and conduct extensive experiments assessing VSP versus five baselines for the three tasks against three LLMs and two datasets. Results show substantial superiority of our CoT-inspired prompting (553.3%, 36.5%, and 30.8% higher F1 accuracy for vulnerability identification, discovery, and patching, respectively, on CVE datasets) over the baselines. Through in-depth case studies analyzing VSP failures, we also reveal current gaps in LLM/CoT for challenging vulnerability cases, while proposing and validating respective improvements.

Figures (6)

• Figure 1: Illustrations of vulnerability semantics (a) and VSP exemplars for the three vulnerability analysis tasks (b,c,d).
• Figure 2: An overview of vulnerability-semantics-guided prompting for vulnerability analysis tasks.
• Figure 3: An example of GPT-3.5 identifying a CWE-476 vulnerability with VSP but not with standard prompting.
• Figure 4: The exemplar corresponding to Figure \ref{['fig:illustrate']} (c) involving unnecessary text (marked as gray) in the reasoning.
• Figure 5: An example that telling the model the contexts by adding comments helps the model do correct analysis.