Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization

TL;DR

This work undertakes a large-scale, empirical evaluation of four off-the-shelf LLMs across twelve prompting techniques to support SSVC-based vulnerability prioritization. By analyzing 384 real-world CVEs from VulZoo/Vulnrichment with three independent trials, the study shows Gemini provides the strongest SDP accuracy and closest SDO alignment, particularly when paired with exemplar-based prompting like Chain-of-Thought. The findings reveal that exemplar-rich prompts substantially improve performance, while Mission & Wellbeing remains the most challenging due to context dependence and synthetic stand-ins. The results suggest LLMs can meaningfully augment vulnerability triage workflows to improve analyst efficiency, but cannot yet replace expert judgment, especially for context-sensitive decisions; further work on calibration, context encoding, and retrieval-augmented approaches could enhance reliability.

Abstract

Security analysts face increasing pressure to triage large and complex vulnerability backlogs. Large Language Models (LLMs) offer a potential aid by automating parts of the interpretation process. We evaluate four models (ChatGPT, Claude, Gemini, and DeepSeek) across twelve prompting techniques to interpret semi-structured and unstructured vulnerability information. As a concrete use case, we test each model's ability to predict decision points in the Stakeholder-Specific Vulnerability Categorization (SSVC) framework: Exploitation, Automatable, Technical Impact, and Mission and Wellbeing. Using 384 real-world vulnerabilities from the VulZoo dataset, we issued more than 165,000 queries to assess performance under prompting styles including one-shot, few-shot, and chain-of-thought. We report F1 scores for each SSVC decision point and Cohen's kappa (weighted and unweighted) for the final SSVC decision outcomes. Gemini consistently ranked highest, leading on three of four decision points and yielding the most correct recommendations. Prompting with exemplars generally improved accuracy, although all models struggled on some decision points. Only DeepSeek achieved fair agreement under weighted metrics, and all models tended to over-predict risk. Overall, current LLMs do not replace expert judgment. However, specific LLM and prompt combinations show moderate effectiveness for targeted SSVC decisions. When applied with care, LLMs can support vulnerability prioritization workflows and help security teams respond more efficiently to emerging threats.

Figures (8)

• Figure 1: Overview of the vulnerability data collection, processing, prompting, and evaluation pipeline. 384 vulnerabilities were selected that were common to both the VulZoo and Vulnrichment datasets. Samples were parsed with the PTs and MWSs, and queried against the LLMs across three trials. Performances were evaluated by: (1) F1-Scores, comparing LLM SDP responses against Vulnrichment SDP values, with a harmonic mean calculated across the trials for each LLM-PT-SDP combination; and (2) Cohen's Kappa, comparing SDOs generated by parsing LLM SDPs and Vulnrichment SDPs through the SSVC decision tree for each MWS.
• Figure 2: Calculation of number of queries per CVE. When multiplied by 384 CVEs, with 12 PTs, and 3 different MWSs, we queried 4 distinct LLMs with 165,888 queries via OpenRouter API.
• Figure 3: Top five performing PT and SDP combinations per LLM. See Appendix \ref{['app:harmonic_means']} for the full dataset.
• Figure 4: SDOs for the High MWS. SDOs of less than 1% omitted. See Appendix \ref{['app:decision_tree_outcome']} for the full dataset.
• Figure 5: Unweighted and Weighted Cohen's Kappa scores for agreement between LLM SSVC decision outcomes and Vulnrichment ground truth SSVC decision outcomes for a High Mission & Wellbeing stand-ins (MWS).