SastBench: A Benchmark for Testing Agentic SAST Triage

TL;DR

SastBench is introduced, a benchmark for evaluating SAST triage agents that combines real CVEs as true positives with filtered SAST tool findings as approximate false positives and features an agent-agnostic design.

Abstract

SAST (Static Application Security Testing) tools are among the most widely used techniques in defensive cybersecurity, employed by commercial and non-commercial organizations to identify potential vulnerabilities in software. Despite their great utility, they generate numerous false positives, requiring costly manual filtering (aka triage). While LLM-powered agents show promise for automating cybersecurity tasks, existing benchmarks fail to emulate real-world SAST finding distributions. We introduce SastBench, a benchmark for evaluating SAST triage agents that combines real CVEs as true positives with filtered SAST tool findings as approximate false positives. SastBench features an agent-agnostic design. We evaluate different agents on the benchmark and present a comparative analysis of their performance, provide a detailed analysis of the dataset, and discuss the implications for future development.

Figures (6)

• Figure 1: Precision-Recall space visualization. Points represent model-agent configurations, with position indicating trade-offs between false positive reduction (precision) and vulnerability detection (recall). Upper-right region represents ideal performance. Simple ReAct agents are indicated by circles, while other variants are presented as diamonds. To avoid clutter, we keep only the important instances.
• Figure 2: Upset plot for the Simple ReAct agent with different LLMs
• Figure 3: Analysis Workflow for CVE-2025-54793
• Figure 4: Visualizations of semantic embedding separability
• Figure 5: Distribution of security findings across programming languages. The multi-language coverage enables assessment of model capabilities across different syntax paradigms and vulnerability contexts.