Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

TL;DR

This study addresses the persistent problem of false positives in SAST by evaluating three LLM-based agent frameworks (Aider, OpenHands, SWE-agent) across the OWASP Benchmark and Vul4J real-world Java vulnerabilities. By enabling iterative reasoning, tool use, and environment interaction, the agents can reduce FP noise dramatically—down to as low as $6.3\%$ FP rate on the benchmark and up to $93.3\%$ FP identification in real-world CodeQL alerts—though gains depend on the backbone model and vulnerability category. The work provides a comparative, multi-model, and multi-target analysis that highlights when agentic approaches outperform vanilla prompting and when they may cause TP suppression or incur higher cost. Practically, the findings suggest deploying LLM-based agents as decision-support tools with careful backbone selection and CWE-aware policies to balance noise reduction against the risk of missing real vulnerabilities and operational costs. The study also offers empirical guidance and identifies promising directions for CWE-aware adaptive strategies and human-in-the-loop auditing to maximize actionable FP filtering in diverse software security contexts.

Abstract

Static Application Security Testing (SAST) tools are essential for identifying software vulnerabilities, but they often produce a high volume of false positives (FPs), imposing a substantial manual triage burden on developers. Recent advances in Large Language Model (LLM) agents offer a promising direction by enabling iterative reasoning, tool use, and environment interaction to refine SAST alerts. However, the comparative effectiveness of different LLM-based agent architectures for FP filtering remains poorly understood. In this paper, we present a comparative study of three state-of-the-art LLM-based agent frameworks, i.e., Aider, OpenHands, and SWE-agent, for vulnerability FP filtering. We evaluate these frameworks using the vulnerabilities from the OWASP Benchmark and real-world open-source Java projects. The experimental results show that LLM-based agents can remove the majority of SAST noise, reducing an initial FP detection rate of over 92% on the OWASP Benchmark to as low as 6.3% in the best configuration. On real-world dataset, the best configuration of LLM-based agents can achieve an FP identification rate of up to 93.3% involving CodeQL alerts. However, the benefits of agents are strongly backbone- and CWE-dependent: agentic frameworks significantly outperform vanilla prompting for stronger models such as Claude Sonnet 4 and GPT-5, but yield limited or inconsistent gains for weaker backbones. Moreover, aggressive FP reduction can come at the cost of suppressing true vulnerabilities, highlighting important trade-offs. Finally, we observe large disparities in computational cost across agent frameworks. Overall, our study demonstrates that LLM-based agents are a powerful but non-uniform solution for SAST FP filtering, and that their practical deployment requires careful consideration of agent design, backbone model choice, vulnerability category, and operational cost.

Figures (6)

• Figure 1: Overview of our work
• Figure 2: Distribution of CWE identified in the Vul4J dataset.
• Figure 3: Distribution of CodeQL rule IDs within the Vul4J sample used for RQ2· ($n=50$).
• Figure 4: Performance comparison of four SAST tools across various CWEs. (a) Heatmap illustrating accuracy. (b) Heatmap illustrating FPR, where values of 1.00 indicate that tools flag all non-vulnerable instances in those categories.
• Figure 5: Distinct FP cases detected by each tool.