Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities

TL;DR

The paper investigates the effectiveness of diverse LLMs in one-shot vulnerability patching using PoV-based validation across real and artificially generated vulnerabilities. Leveraging the Vul4J dataset and 14 LLMs, it shows real vulnerabilities are patched more readily than artificial ones, with significant cross-model overlap but scarce complementary patches. The study highlights limitations of current LLM generalization to artificial cases and discusses implications for ensemble strategies and prompting. Overall, it provides a rigorous, execution-based assessment of LLM-driven patch generation and outlines concrete directions for improving generalization and reliability.

Abstract

Automated vulnerability patching is crucial for software security, and recent advancements in Large Language Models (LLMs) present promising capabilities for automating this task. However, existing research has primarily assessed LLMs using publicly disclosed vulnerabilities, leaving their effectiveness on related artificial vulnerabilities largely unexplored. In this study, we empirically evaluate the patching effectiveness and complementarity of several prominent LLMs, such as OpenAI's GPT variants, LLaMA, DeepSeek, and Mistral models, using both real and artificial vulnerabilities. Our evaluation employs Proof-of-Vulnerability (PoV) test execution to concretely assess whether LLM-generated source code successfully patches vulnerabilities. Our results reveal that LLMs patch real vulnerabilities more effectively compared to artificial ones. Additionally, our analysis reveals significant variability across LLMs in terms of overlapping (multiple LLMs patching the same vulnerabilities) and complementarity (vulnerabilities patched exclusively by a single LLM), emphasizing the importance of selecting appropriate LLMs for effective vulnerability patching.

Figures (4)

• Figure 1: Vulnerability CVE-2018-17201 (Fig. \ref{['fig_example_1_a']}) causes an infinite loop that can make the program hang, enabling a Denial-of-Service (DoS) attack. This issue is addressed by introducing a conditional exception using an "if" statement (Fig. \ref{['fig_example_1_b']}). However, the code-modification (Fig. \ref{['fig_example_1_c']}) alters the "if" condition, effectively nullifying the fix and reintroduces the vulnerability.
• Figure 2: Vulnerability CVE-2018-1000850 (Fig. \ref{['fig_example_2_a']}) enables path traversal and access to restricted directories. The patch adds a conditional check for '.' or '..' in "newRelativeUrl" (Fig. \ref{['fig_example_2_b']}). The code-modification (Fig. \ref{['fig_example_2_c']}) replaces "newRelativeUrl" with "name", nullifying the fix and reintroducing the vulnerability.
• Figure 3: Patching effectiveness of LLMs for real and artificial vulnerabilities.
• Figure 4: Overlapping and complementarity of LLMs in patching real and artificial vulnerabilities