Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models

Romy Fieblinger, Md Tanvirul Alam, Nidhi Rastogi

TL;DR

The paper investigates automating actionable Cyber Threat Intelligence (CTI) extraction from unstructured CTI using Large Language Models (LLMs) and Knowledge Graphs (KGs). It evaluates open-source LLMs (Llama 2, Mistral 7B Instruct, Zephyr) across few-shot prompting, guidance, and fine-tuning to generate structured triples and construct a KG, followed by link prediction experiments. Key findings show that guidance and fine-tuning outperform sole prompt engineering, and incorporating an ontology and entity typing improves triple quality, though large-scale data introduces substantial noise and scaling challenges. The work demonstrates a viable path to automated CTI processing with practical impact on threat understanding and response, while outlining significant future work to achieve robust, scalable deployment.

Abstract

Cyber threats are constantly evolving. Extracting actionable insights from unstructured Cyber Threat Intelligence (CTI) data is essential to guide cybersecurity decisions. Increasingly, organizations like Microsoft, Trend Micro, and CrowdStrike are using generative AI to facilitate CTI extraction. This paper addresses the challenge of automating the extraction of actionable CTI using advancements in Large Language Models (LLMs) and Knowledge Graphs (KGs). We explore the application of state-of-the-art open-source LLMs, including the Llama 2 series, Mistral 7B Instruct, and Zephyr for extracting meaningful triples from CTI texts. Our methodology evaluates techniques such as prompt engineering, the guidance framework, and fine-tuning to optimize information extraction and structuring. The extracted data is then utilized to construct a KG, offering a structured and queryable representation of threat intelligence. Experimental results demonstrate the effectiveness of our approach in extracting relevant information, with guidance and fine-tuning showing superior performance over prompt engineering. However, while our methods prove effective in small-scale tests, applying LLMs to large-scale data for KG construction and Link Prediction presents ongoing challenges.

Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models

TL;DR

The paper investigates automating actionable Cyber Threat Intelligence (CTI) extraction from unstructured CTI using Large Language Models (LLMs) and Knowledge Graphs (KGs). It evaluates open-source LLMs (Llama 2, Mistral 7B Instruct, Zephyr) across few-shot prompting, guidance, and fine-tuning to generate structured triples and construct a KG, followed by link prediction experiments. Key findings show that guidance and fine-tuning outperform sole prompt engineering, and incorporating an ontology and entity typing improves triple quality, though large-scale data introduces substantial noise and scaling challenges. The work demonstrates a viable path to automated CTI processing with practical impact on threat understanding and response, while outlining significant future work to achieve robust, scalable deployment.

Abstract

Cyber threats are constantly evolving. Extracting actionable insights from unstructured Cyber Threat Intelligence (CTI) data is essential to guide cybersecurity decisions. Increasingly, organizations like Microsoft, Trend Micro, and CrowdStrike are using generative AI to facilitate CTI extraction. This paper addresses the challenge of automating the extraction of actionable CTI using advancements in Large Language Models (LLMs) and Knowledge Graphs (KGs). We explore the application of state-of-the-art open-source LLMs, including the Llama 2 series, Mistral 7B Instruct, and Zephyr for extracting meaningful triples from CTI texts. Our methodology evaluates techniques such as prompt engineering, the guidance framework, and fine-tuning to optimize information extraction and structuring. The extracted data is then utilized to construct a KG, offering a structured and queryable representation of threat intelligence. Experimental results demonstrate the effectiveness of our approach in extracting relevant information, with guidance and fine-tuning showing superior performance over prompt engineering. However, while our methods prove effective in small-scale tests, applying LLMs to large-scale data for KG construction and Link Prediction presents ongoing challenges.
Paper Structure (23 sections, 3 equations, 1 figure, 9 tables)

This paper contains 23 sections, 3 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Approach
  4. Research Questions
  5. Pretrained Models
  6. Dataset Generation
  7. Information Extraction Methods using LLM
  8. Few-Shot Setting
  9. Prompt Engineering
  10. Guidance
  11. Fine-tuning
  12. Inference & Decoding Strategies
  13. Implementation Details
  14. Evaluation Metrics
  15. Experiment and Results
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Proposed approach: Outline for LLM-based CTI extraction and KG development. Initial stages involve adapting the models with Few-Shot Prompting and Fine-tuning to generate triple output. Following this, extensive evaluation determines the best model and prompt combination. The top models are then utilized for triple generation in the KG, leading to Link Prediction on the optimal KG, showcasing the transformation from raw data to actionable intelligence.