Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CVE-LLM : Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models

Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, Oladimeji Farri

TL;DR

The paper presents CVE-LLM, an ontology-assisted LLM designed to automate vulnerability evaluations for medical devices by learning from historical SHS assessments and integrating CSKG enrichment to link CVE-CWE-CAPEC knowledge. It adopts a pretrain-then-finetune pipeline (DAPT on MPT-7B with vocabulary expansion, followed by SFT on 750K instructions) and targets outputs including VEXCategory, VEXJustification, InternalComment, and Vector, with deployment through a CSMS and a human-in-the-loop validation. Empirical results show CVE-LLM outperforms comparable open-source LLMs on key outputs, and ablations demonstrate substantial gains from domain adaptation and ontology enrichment, along with significant inference-time speedups via vLLM and batching. The work highlights practical impact for scaling vulnerability evaluations in large commercial portfolios, while acknowledging challenges in ontology coverage and hallucinations, and outlining directions for broader knowledge integration and improved robustness.

Abstract

The National Vulnerability Database (NVD) publishes over a thousand new vulnerabilities monthly, with a projected 25 percent increase in 2024, highlighting the crucial need for rapid vulnerability identification to mitigate cybersecurity attacks and save costs and resources. In this work, we propose using large language models (LLMs) to learn vulnerability evaluation from historical assessments of medical device vulnerabilities in a single manufacturer's portfolio. We highlight the effectiveness and challenges of using LLMs for automatic vulnerability evaluation and introduce a method to enrich historical data with cybersecurity ontologies, enabling the system to understand new vulnerabilities without retraining the LLM. Our LLM system integrates with the in-house application - Cybersecurity Management System (CSMS) - to help Siemens Healthineers (SHS) product cybersecurity experts efficiently assess the vulnerabilities in our products. Also, we present guidelines for efficient integration of LLMs into the cybersecurity tool.

CVE-LLM : Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models

TL;DR

The paper presents CVE-LLM, an ontology-assisted LLM designed to automate vulnerability evaluations for medical devices by learning from historical SHS assessments and integrating CSKG enrichment to link CVE-CWE-CAPEC knowledge. It adopts a pretrain-then-finetune pipeline (DAPT on MPT-7B with vocabulary expansion, followed by SFT on 750K instructions) and targets outputs including VEXCategory, VEXJustification, InternalComment, and Vector, with deployment through a CSMS and a human-in-the-loop validation. Empirical results show CVE-LLM outperforms comparable open-source LLMs on key outputs, and ablations demonstrate substantial gains from domain adaptation and ontology enrichment, along with significant inference-time speedups via vLLM and batching. The work highlights practical impact for scaling vulnerability evaluations in large commercial portfolios, while acknowledging challenges in ontology coverage and hallucinations, and outlining directions for broader knowledge integration and improved robustness.

Abstract

The National Vulnerability Database (NVD) publishes over a thousand new vulnerabilities monthly, with a projected 25 percent increase in 2024, highlighting the crucial need for rapid vulnerability identification to mitigate cybersecurity attacks and save costs and resources. In this work, we propose using large language models (LLMs) to learn vulnerability evaluation from historical assessments of medical device vulnerabilities in a single manufacturer's portfolio. We highlight the effectiveness and challenges of using LLMs for automatic vulnerability evaluation and introduce a method to enrich historical data with cybersecurity ontologies, enabling the system to understand new vulnerabilities without retraining the LLM. Our LLM system integrates with the in-house application - Cybersecurity Management System (CSMS) - to help Siemens Healthineers (SHS) product cybersecurity experts efficiently assess the vulnerabilities in our products. Also, we present guidelines for efficient integration of LLMs into the cybersecurity tool.

Paper Structure

This paper contains 29 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. DAPT Dataset
  4. SFT Dataset
  5. Enriched Instructions Through CSKG
  6. Formation of SFT Dataset
  7. Model Training
  8. DAPT Model Training
  9. SFT Model Training
  10. Deployment
  11. Generation of Evaluations
  12. Rule-Based Correction
  13. Model Result Integration Into CSMS System
  14. Results
  15. Benchmarking With Open Source LLMs
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Schematic overview of CVE-LLM
  • Figure 2: Ablation studies. The Evaluation metrics is Rouge-L for InternalComment and CustomerComment, and micro-F1 for the rest.