An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems

TL;DR

This study benchmarks three LLM-based vulnerability-detection strategies—Retrieval-Augmented Generation (RAG), Supervised Fine-Tuning (SFT) with QLoRA adapters, and a Dual-Agent LLM framework—against a baseline model on a balanced dataset covering five CWE categories. RAG achieves the highest overall accuracy (0.86) and F1 (0.85), highlighting the value of external domain knowledge; SFT and the Dual-Agent approach also offer notable gains while reducing resource demands and improving reasoning reliability. The authors contribute a balanced, expert-annotated dataset drawn from BigVul and enterprise code to address common data gaps and demonstrate statistically significant improvements over the baseline. The work supports practical deployment guidance for vulnerability detection, showing that domain-aware augmentation and multi-agent validation can meaningfully improve vulnerability detection in real-world codebases.

Abstract

The rapid advancement of Large Language Models (LLMs) presents new opportunities for automated software vulnerability detection, a crucial task in securing modern codebases. This paper presents a comparative study on the effectiveness of LLM-based techniques for detecting software vulnerabilities. The study evaluates three approaches, Retrieval-Augmented Generation (RAG), Supervised Fine-Tuning (SFT), and a Dual-Agent LLM framework, against a baseline LLM model. A curated dataset was compiled from Big-Vul and real-world code repositories from GitHub, focusing on five critical Common Weakness Enumeration (CWE) categories: CWE-119, CWE-399, CWE-264, CWE-20, and CWE-200. Our RAG approach, which integrated external domain knowledge from the internet and the MITRE CWE database, achieved the highest overall accuracy (0.86) and F1 score (0.85), highlighting the value of contextual augmentation. Our SFT approach, implemented using parameter-efficient QLoRA adapters, also demonstrated strong performance. Our Dual-Agent system, an architecture in which a secondary agent audits and refines the output of the first, showed promise in improving reasoning transparency and error mitigation, with reduced resource overhead. These results emphasize that incorporating a domain expertise mechanism significantly strengthens the practical applicability of LLMs in real-world vulnerability detection tasks.

Figures (6)

• Figure 1: Instructional prompt for the LLM Classification task.
• Figure 2: Retrieval-Augmentation Generation Process
• Figure 3: Supervised Fine-Tuning Process
• Figure 4: Dual-Agent LLM System Overview
• Figure 5: F1-Score Comparison Among Techniques