Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

TL;DR

These findings demonstrate that LLM security patches require rigorous validation before deployment, and the dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies.

Abstract

Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.

Figures (2)

• Figure 1: Experimental pipeline: vulnerable code $\rightarrow$ LLM $\rightarrow$ patch extraction $\rightarrow$ tri-axis evaluation.
• Figure 2: SRS distribution showing bimodal pattern: clusters at SRS$\approx$1.0 (perfect) and SRS$\approx$0.5 (functional but insecure).