On Technique Identification and Threat-Actor Attribution using LLMs and Embedding Models

TL;DR

This paper tackles the problem of attributing cyber-attacks by converting unstructured threat reports into structured TTP indicators and using them to predict threat actors. It compares off-the-shelf LLMs (GPT-4) against vector-embedding searches to extract TTPs, then builds an end-to-end pipeline that uses a probabilistic weighting scheme and Bayes-like reasoning to attribute incidents to MITRE ATT&CK groups. Key findings show that GPT-4 produces TTPs with substantial noise and low overlap with human data, but these TTPs preserve frequency patterns similar to MITRE and can train models that outperform baseline attribution. A vector-embedding–based approach (VE) provides more transparent, probabilistic outputs and, when paired with expert priors or HyDE augmentation, yields the best attribution performance (average rank around 7.55 out of 29), illustrating the potential and limitations of LLM-driven CTI pipelines for decision support in threat intelligence. The work highlights that human-in-the-loop guidance remains essential and points to future directions in fine-tuning, dynamic signal integration, and improved interpretability for operational use.

Abstract

Attribution of cyber-attacks remains a complex but critical challenge for cyber defenders. Currently, manual extraction of behavioral indicators from dense forensic documentation causes significant attribution delays, especially following major incidents at the international scale. This research evaluates large language models (LLMs) for cyber-attack attribution based on behavioral indicators extracted from forensic documentation. We test OpenAI's GPT-4 and text-embedding-3-large for identifying threat actors' tactics, techniques, and procedures (TTPs) by comparing LLM-generated TTPs against human-generated data from MITRE ATT&CK Groups. Our framework then identifies TTPs from text using vector embedding search and builds profiles to attribute new attacks for a machine learning model to learn. Key contributions include: (1) assessing off-the-shelf LLMs for TTP extraction and attribution, and (2) developing an end-to-end pipeline from raw CTI documents to threat-actor prediction. This research finds that standard LLMs generate TTP datasets with noise, resulting in a low similarity to human-generated datasets. However, the TTPs generated are similar in frequency to those within the existing MITRE datasets. Additionally, although these TTPs are different than human-generated datasets, our work demonstrates that they still prove useful for training a model that performs above baseline on attribution. Project code and files are contained here: https://github.com/kylag/ttp_attribution.

Figures (4)

• Figure 1: GPT prompt used to predict TTPs from an attached document.
• Figure 2: Testing split using unseen 10% of total documentation.
• Figure 3: TTP frequencies from MITRE and GPT-4 generated datasets.
• Figure 4: Comparing Jaccard similarity for the exhaustive baseline, the vector embeddings (VE), and GPT-4.