The Semantic Trap: Do Fine-tuned LLMs Learn Vulnerability Root Cause or Just Functional Pattern?

TL;DR

Large language models show promise in vulnerability detection after fine-tuning, but it is unclear whether they learn vulnerability root causes or rely on functional patterns. The authors introduce TrapEval, a framework combining paired (V2P) and unpaired (V2N) datasets, semantic-preserving perturbations, and CodeBLEU-based semantic-gap analysis to probe true security reasoning. Across five state-of-the-art LLMs, results reveal that fine-tuning improves standard metrics yet models struggle to distinguish vulnerabilities from patches, degrade under minor semantic edits, and rely on code-domain patterns when the semantic gap is small. The work highlights a semantic-trap in current training regimes and argues for code-semantics-aware fine-tuning and robust cross-dataset evaluation to enable trustworthy LLM-assisted software security analysis.

Abstract

LLMs demonstrate promising performance in software vulnerability detection after fine-tuning. However, it remains unclear whether these gains reflect a genuine understanding of vulnerability root causes or merely an exploitation of functional patterns. In this paper, we identify a critical failure mode termed the "semantic trap," where fine-tuned LLMs achieve high detection scores by associating certain functional domains with vulnerability likelihood rather than reasoning about the underlying security semantics.To systematically evaluate this phenomenon, we propose TrapEval, a comprehensive evaluation framework designed to disentangle vulnerability root cause from functional pattern. TrapEval introduces two complementary datasets derived from real-world open-source projects: V2N, which pairs vulnerable code with unrelated benign code, and V2P, which pairs vulnerable code with its corresponding patched version, forcing models to distinguish near-identical code that differs only in subtle security-critical logic. Using TrapEval, we fine-tune five representative state-of-the-art LLMs across three model families and evaluate them under cross-dataset testing, semantic-preserving perturbations, and varying degrees of semantic gap measured by CodeBLEU.Our empirical results reveal that, despite improvements in metrics, fine-tuned LLMs consistently struggle to distinguish vulnerable code from its patched counterpart, exhibit severe robustness degradation under minor semantic-preserving transformations, and rely heavily on functional-context shortcuts when the semantic gap is small. These findings provide strong evidence that current fine-tuning practices often fail to impart true vulnerability reasoning. Our findings serve as a wake-up call: high benchmark scores on traditional datasets may be illusory, masking the model's inability to understand the true causal logic of vulnerabilities.

Figures (2)

• Figure 1: Overview of TrapEval.
• Figure 2: Prompt Template during fine-tuning and evaluation.