Towards Effective Identification of Attack Techniques in Cyber Threat Intelligence Reports using Large Language Models

TL;DR

This work tackles the problem of extracting MITRE ATT&CK techniques from web CTI reports, a task hampered by domain complexity, verbosity, class imbalance, and overfitting. The authors compare four configurations including TRAM and vanilla Llama2 models and propose a two-step pipeline in which GPT-3.5 summarises CTI reports and a SciBERT model retrained on a rebalanced, LLM-augmented dataset performs the final classification. The main finding is that standalone LLMs underperform, while the summarisation-plus-retraining approach yields substantial improvements, with F1-scores surpassing 0.90 on several techniques. The work contributes a practical, collaborative CTI extraction workflow and demonstrates the value of human-AI collaboration for scalable threat intelligence. This lays groundwork for integrating web-based CTI systems into SOC operations.

Abstract

This work evaluates the performance of Cyber Threat Intelligence (CTI) extraction methods in identifying attack techniques from threat reports available on the web using the MITRE ATT&CK framework. We analyse four configurations utilising state-of-the-art tools, including the Threat Report ATT&CK Mapper (TRAM) and open-source Large Language Models (LLMs) such as Llama2. Our findings reveal significant challenges, including class imbalance, overfitting, and domain-specific complexity, which impede accurate technique extraction. To mitigate these issues, we propose a novel two-step pipeline: first, an LLM summarises the reports, and second, a retrained SciBERT model processes a rebalanced dataset augmented with LLM-generated data. This approach achieves an improvement in F1-scores compared to baseline models, with several attack techniques surpassing an F1-score of 0.90. Our contributions enhance the efficiency of web-based CTI systems and support collaborative cybersecurity operations in an interconnected digital landscape, paving the way for future research on integrating human-AI collaboration platforms.

Figures (2)

• Figure 1: Our evaluation methodology uses four configurations.
• Figure 2: Performance of TRAM using different Configurations proposed in our work on ATD dataset.