Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proactively Detecting Threats: A Novel Approach Using LLMs

Aniesh Chawla, Udbhav Prasad

TL;DR

The paper tackles the challenge of proactively detecting indicators of compromise (IOCs) by leveraging large language models (LLMs) to classify IOCs from unstructured threat intelligence sources. It introduces an automated end-to-end pipeline that aggregates threat reports from 15 sources, extracts IOCs with regex and MITRE ATT&CK enrichment, and evaluates six LLMs on 479 pages, demonstrating that Gemini 1.5 Pro achieves high precision (0.958) and perfect malicious recall (1.0) with reasonable specificity (0.788). The study reveals strong cross-model trade-offs: while some models excel at recalling actual threats, others better distinguish benign indicators, with significant challenges in domain-type indicators and benign-context misclassification. The contributions include a scalable multi-source IOC collection and a comparative LLM evaluation across architectures, highlighting the potential and current limitations of proactive LLM-based threat detection in enterprise security. The work lays a foundation for automated threat-intelligence processing and prompts further expansion to richer indicator types and document formats to meaningfully advance proactive cybersecurity tooling.

Abstract

Enterprise security faces escalating threats from sophisticated malware, compounded by expanding digital operations. This paper presents the first systematic evaluation of large language models (LLMs) to proactively identify indicators of compromise (IOCs) from unstructured web-based threat intelligence sources, distinguishing it from reactive malware detection approaches. We developed an automated system that pulls IOCs from 15 web-based threat report sources to evaluate six LLM models (Gemini, Qwen, and Llama variants). Our evaluation of 479 webpages containing 2,658 IOCs (711 IPv4 addresses, 502 IPv6 addresses, 1,445 domains) reveals significant performance variations. Gemini 1.5 Pro achieved 0.958 precision and 0.788 specificity for malicious IOC identification, while demonstrating perfect recall (1.0) for actual threats.

Proactively Detecting Threats: A Novel Approach Using LLMs

TL;DR

The paper tackles the challenge of proactively detecting indicators of compromise (IOCs) by leveraging large language models (LLMs) to classify IOCs from unstructured threat intelligence sources. It introduces an automated end-to-end pipeline that aggregates threat reports from 15 sources, extracts IOCs with regex and MITRE ATT&CK enrichment, and evaluates six LLMs on 479 pages, demonstrating that Gemini 1.5 Pro achieves high precision (0.958) and perfect malicious recall (1.0) with reasonable specificity (0.788). The study reveals strong cross-model trade-offs: while some models excel at recalling actual threats, others better distinguish benign indicators, with significant challenges in domain-type indicators and benign-context misclassification. The contributions include a scalable multi-source IOC collection and a comparative LLM evaluation across architectures, highlighting the potential and current limitations of proactive LLM-based threat detection in enterprise security. The work lays a foundation for automated threat-intelligence processing and prompts further expansion to richer indicator types and document formats to meaningfully advance proactive cybersecurity tooling.

Abstract

Enterprise security faces escalating threats from sophisticated malware, compounded by expanding digital operations. This paper presents the first systematic evaluation of large language models (LLMs) to proactively identify indicators of compromise (IOCs) from unstructured web-based threat intelligence sources, distinguishing it from reactive malware detection approaches. We developed an automated system that pulls IOCs from 15 web-based threat report sources to evaluate six LLM models (Gemini, Qwen, and Llama variants). Our evaluation of 479 webpages containing 2,658 IOCs (711 IPv4 addresses, 502 IPv6 addresses, 1,445 domains) reveals significant performance variations. Gemini 1.5 Pro achieved 0.958 precision and 0.788 specificity for malicious IOC identification, while demonstrating perfect recall (1.0) for actual threats.
Paper Structure (16 sections, 3 figures, 3 tables)

This paper contains 16 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Datasets and Evaluation Frameworks
  4. Specialized Threat Detection Algorithms
  5. Traditional Machine Learning Approaches
  6. Research Gaps and Our Contribution
  7. Data Collection
  8. Threat Intelligence Sources & Analysis
  9. Threat Intelligence Sharing Formats
  10. The role of AI and Automation
  11. Threat Intelligence News Sources
  12. Web-crawler and Parser
  13. Performance of LLMs in IOC Classification
  14. When Models Misidentify Benign Data
  15. Discussion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: System to Evaluate threat indicators from the web
  • Figure 2: Examples of different IOC data representations (side-by-side) abusech
  • Figure 3: IP address that looks malicious from context abusech2011.