Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation

TL;DR

This work investigates how Large Language Models can interpret MITRE ATT&CK TTP descriptions by comparing supervised fine-tuning of encoder-only models with retrieval-augmented generation for decoder-only models. It demonstrates that decoder-only LLMs augmented with RAG outperform encoder-only models in tactic interpretation, achieving high recall but facing precision challenges, and reveals that generic prompts can sometimes outperform highly tailored prompts. Using MITRE ATT&CK v14.1 data and FAISS-based retrieval, the study provides practical guidance on deploying context-aware LLMs for cyber threat intelligence without fine-tuning large models. The findings highlight an important recall-precision trade-off and propose directions for improving precision in RAG-enabled, generation-based cyber-ops tools.

Abstract

Tactics, Techniques, and Procedures (TTPs) outline the methods attackers use to exploit vulnerabilities. The interpretation of TTPs in the MITRE ATT&CK framework can be challenging for cybersecurity practitioners due to presumed expertise and complex dependencies. Meanwhile, advancements with Large Language Models (LLMs) have led to recent surge in studies exploring its uses in cybersecurity operations. It is, however, unclear how LLMs can be used in an efficient and proper way to provide accurate responses for critical domains such as cybersecurity. This leads us to investigate how to better use two types of LLMs: small-scale encoder-only (e.g., RoBERTa) and larger decoder-only (e.g., GPT-3.5) LLMs to comprehend and summarize TTPs with the intended purposes (i.e., tactics) of a cyberattack procedure. This work studies and compares the uses of supervised fine-tuning (SFT) of encoder-only LLMs vs. Retrieval Augmented Generation (RAG) for decoder-only LLMs (without fine-tuning). Both SFT and RAG techniques presumably enhance the LLMs with relevant contexts for each cyberattack procedure. Our studies show decoder-only LLMs with RAG achieves better performance than encoder-only models with SFT, particularly when directly relevant context is extracted by RAG. The decoder-only results could suffer low `Precision' while achieving high `Recall'. Our findings further highlight a counter-intuitive observation that more generic prompts tend to yield better predictions of cyberattack tactics than those that are more specifically tailored.

Figures (3)

• Figure 1: The pair-wise overlap of tactics on ATT&CK procedure descriptions.
• Figure 2: The process of supervised multi-label fine-tuning of e-LLMs with high-level ATT&CK descriptions (excluding 'procedures') -- adopted from fayyazi2023uses.
• Figure 3: The overall process of RAG with relevant ATT&CK procedures of d-LLMs.