Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Description to Score: Can LLMs Quantify Vulnerabilities?

Sima Jafarikhah, Daniel Thompson, Eva Deans, Hossein Siadati, Yi Liu

TL;DR

Vulnerability scoring is a resource-intensive process hindered by backlog and subjectivity. The authors evaluate six general-purpose LLMs on over 31,000 CVEs to extract CVSS v3.1 base metrics from vulnerability descriptions and compute final scores, with a two-shot prompting strategy and a meta-classifier to fuse model outputs. GPT-5 and Gemini-2.5-Flash show strong performance on several metrics, but minority classes and ambiguous descriptions limit reliability; meta-classification provides modest gains across all metrics. The findings illustrate the potential for scalable automated vulnerability triage while highlighting the need for richer, clearer vulnerability descriptions and external contextual signals to improve robustness in operational settings.

Abstract

Manual vulnerability scoring, such as assigning Common Vulnerability Scoring System (CVSS) scores, is a resource-intensive process that is often influenced by subjective interpretation. This study investigates the potential of general-purpose large language models (LLMs), namely ChatGPT, Llama, Grok, DeepSeek, and Gemini, to automate this process by analyzing over 31{,}000 recent Common Vulnerabilities and Exposures (CVE) entries. The results show that LLMs substantially outperform the baseline on certain metrics (e.g., \textit{Availability Impact}), while offering more modest gains on others (e.g., \textit{Attack Complexity}). Moreover, model performance varies across both LLM families and individual CVSS metrics, with ChatGPT-5 attaining the highest precision. Our analysis reveals that LLMs tend to misclassify many of the same CVEs, and ensemble-based meta-classifiers only marginally improve performance. Further examination shows that CVE descriptions often lack critical context or contain ambiguous phrasing, which contributes to systematic misclassifications. These findings underscore the importance of enhancing vulnerability descriptions and incorporating richer contextual details to support more reliable automated reasoning and alleviate the growing backlog of CVEs awaiting triage.

From Description to Score: Can LLMs Quantify Vulnerabilities?

TL;DR

Vulnerability scoring is a resource-intensive process hindered by backlog and subjectivity. The authors evaluate six general-purpose LLMs on over 31,000 CVEs to extract CVSS v3.1 base metrics from vulnerability descriptions and compute final scores, with a two-shot prompting strategy and a meta-classifier to fuse model outputs. GPT-5 and Gemini-2.5-Flash show strong performance on several metrics, but minority classes and ambiguous descriptions limit reliability; meta-classification provides modest gains across all metrics. The findings illustrate the potential for scalable automated vulnerability triage while highlighting the need for richer, clearer vulnerability descriptions and external contextual signals to improve robustness in operational settings.

Abstract

Manual vulnerability scoring, such as assigning Common Vulnerability Scoring System (CVSS) scores, is a resource-intensive process that is often influenced by subjective interpretation. This study investigates the potential of general-purpose large language models (LLMs), namely ChatGPT, Llama, Grok, DeepSeek, and Gemini, to automate this process by analyzing over 31{,}000 recent Common Vulnerabilities and Exposures (CVE) entries. The results show that LLMs substantially outperform the baseline on certain metrics (e.g., \textit{Availability Impact}), while offering more modest gains on others (e.g., \textit{Attack Complexity}). Moreover, model performance varies across both LLM families and individual CVSS metrics, with ChatGPT-5 attaining the highest precision. Our analysis reveals that LLMs tend to misclassify many of the same CVEs, and ensemble-based meta-classifiers only marginally improve performance. Further examination shows that CVE descriptions often lack critical context or contain ambiguous phrasing, which contributes to systematic misclassifications. These findings underscore the importance of enhancing vulnerability descriptions and incorporating richer contextual details to support more reliable automated reasoning and alleviate the growing backlog of CVEs awaiting triage.

Paper Structure

This paper contains 26 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Common Vulnerability Scoring System
  4. Challenges of Triaging CVEs by NVD
  5. Related Work
  6. Study Design
  7. Objective
  8. Data Collection
  9. Large Language Models
  10. Prompt Engineering
  11. Evaluation Metrics
  12. Analysis
  13. Exploring the CVE Dataset
  14. Analysis of Individual LLM Classifiers
  15. Analysis of the content of "description" field
  16. ...and 11 more sections

Figures (7)

  • Figure 1: CVSS Metric Groups cvssv3.1
  • Figure 2: Distribution of Categories Across CVSS Metrics
  • Figure 3: Distribution of Categories for Severity
  • Figure 4: Correlation of Base Metrics
  • Figure 5: Confusion Matrices for CVSS metrics for GPT-5, the best performing model, per metric
  • ...and 2 more figures