Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

Yuqiao Meng, Luoxi Tang, Feiyang Yu, Jinyuan Jia, Guanhua Yan, Ping Yang, Zhaohan Xi

TL;DR

This work interrogates intrinsic vulnerabilities of LLMs in CTI tasks by systematically evaluating across CTI benchmarks and real threat reports. It introduces a novel triad of methods—stratification, autoregressive refinement, and human-in-the-loop categorization—to reliably classify failure modes. The study identifies three core vulnerabilities—spurious correlations, contradictory knowledge, and constrained generalization—and demonstrates how these weaknesses propagate across CTI stages. The findings yield principled design insights for more robust LLM-powered CTI systems and provide a pathway for safer deployment in security operations.

Abstract

Large Language Models (LLMs) are intensively used to assist security analysts in counteracting the rapid exploitation of cyber threats, wherein LLMs offer cyber threat intelligence (CTI) to support vulnerability assessment and incident response. While recent work has shown that LLMs can support a wide range of CTI tasks such as threat analysis, vulnerability detection, and intrusion defense, significant performance gaps persist in practical deployments. In this paper, we investigate the intrinsic vulnerabilities of LLMs in CTI, focusing on challenges that arise from the nature of the threat landscape itself rather than the model architecture. Using large-scale evaluations across multiple CTI benchmarks and real-world threat reports, we introduce a novel categorization methodology that integrates stratification, autoregressive refinement, and human-in-the-loop supervision to reliably analyze failure instances. Through extensive experiments and human inspections, we reveal three fundamental vulnerabilities: spurious correlations, contradictory knowledge, and constrained generalization, that limit LLMs in effectively supporting CTI. Subsequently, we provide actionable insights for designing more robust LLM-powered CTI systems to facilitate future research.

Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

TL;DR

This work interrogates intrinsic vulnerabilities of LLMs in CTI tasks by systematically evaluating across CTI benchmarks and real threat reports. It introduces a novel triad of methods—stratification, autoregressive refinement, and human-in-the-loop categorization—to reliably classify failure modes. The study identifies three core vulnerabilities—spurious correlations, contradictory knowledge, and constrained generalization—and demonstrates how these weaknesses propagate across CTI stages. The findings yield principled design insights for more robust LLM-powered CTI systems and provide a pathway for safer deployment in security operations.

Abstract

Large Language Models (LLMs) are intensively used to assist security analysts in counteracting the rapid exploitation of cyber threats, wherein LLMs offer cyber threat intelligence (CTI) to support vulnerability assessment and incident response. While recent work has shown that LLMs can support a wide range of CTI tasks such as threat analysis, vulnerability detection, and intrusion defense, significant performance gaps persist in practical deployments. In this paper, we investigate the intrinsic vulnerabilities of LLMs in CTI, focusing on challenges that arise from the nature of the threat landscape itself rather than the model architecture. Using large-scale evaluations across multiple CTI benchmarks and real-world threat reports, we introduce a novel categorization methodology that integrates stratification, autoregressive refinement, and human-in-the-loop supervision to reliably analyze failure instances. Through extensive experiments and human inspections, we reveal three fundamental vulnerabilities: spurious correlations, contradictory knowledge, and constrained generalization, that limit LLMs in effectively supporting CTI. Subsequently, we provide actionable insights for designing more robust LLM-powered CTI systems to facilitate future research.

Paper Structure

This paper contains 33 sections, 1 equation, 4 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background: cyber threat intelligence stages
  4. Motivation: LLMs remain insufficient in variant CTI tasks
  5. Methodology: Categorize Failure Instances
  6. Stratifying instances without hard-label annotations (RQ$_1$)
  7. Autoregressive failure mode determination (RQ$_2$)
  8. Human-in-the-loop categorization of failure instances (RQ$_3$)
  9. Research Finding
  10. Varying Performance of LLMs (RQ$_4$)
  11. Root Causes of Vulnerabilities (RQ$_5$)
  12. Study of Intertwined Vulnerabilities (RQ$_6$)
  13. Related Work
  14. Conclusion
  15. Complementary Details of CTI
  16. ...and 18 more sections

Figures (4)

  • Figure 1: (Left) Failure ratios of cybersecurity agents. (Right) Examples of vulnerabilities.
  • Figure 2: Summarization of the vulnerability types of LLMs in various CTI stages. Ratios are calculated over the entire dataset (§\ref{['ssec:eval']}). Vulnerabilities may overlap within a single threat instance.
  • Figure 3: Overview of method to categorize failure instances (addressing RQ$_1$-RQ$_3$).
  • Figure 4: Varying proportions of vulnerabilities (types listed in Figure \ref{['fig:summary']}). Note that different vulnerabilities can intertwine within the same instance, which is particularly common in Contradictory Knowledge (CK) and Constrained Generalization (CG), less common in Spurious Correlation (SC).