Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instantiating Standards: Enabling Standard-Driven Text TTP Extraction with Evolvable Memory

Cheng Meng, ZhengWei Jiang, QiuYun Wang, XinYi Li, ChunYan Ma, FangMing Dong, FangLi Ren, BaoXu Liu

TL;DR

The paper tackles the challenge of extracting MITRE ATT&CK TTPs from unstructured CTI text while ensuring adherence to the official standard and providing transparency. It introduces a dual-layer Situational Knowledge Representation (SKR) paired with an Evolvable Memory System, both managed by an LLM, to generate, update, and apply contextual knowledge for TTP classification. A two-stage extraction pipeline (Stage1 retrieval/classification and Stage2 refinement/verification) enables accurate disambiguation and interoperability with other extraction systems, supported by memory generation, optimization, and forgetting mechanisms. Empirical results on the nguyen-etal-2024-noise dataset show substantial performance gains and improved standardization and explainability, with an $11\%$ higher F1 score than GPT-4o on procedures data, marking the first work to generate and apply new knowledge for TTP extraction using LLMs.

Abstract

Extracting MITRE ATT\&CK Tactics, Techniques, and Procedures (TTPs) from natural language threat reports is crucial yet challenging. Existing methods primarily focus on performance metrics using data-driven approaches, often neglecting mechanisms to ensure faithful adherence to the official standard. This deficiency compromises reliability and consistency of TTP assignments, creating intelligence silos and contradictory threat assessments across organizations. To address this, we introduce a novel framework that converts abstract standard definitions into actionable, contextualized knowledge. Our method utilizes Large Language Model (LLM) to generate, update, and apply this knowledge. This framework populates an evolvable memory with dual-layer situational knowledge instances derived from labeled examples and official definitions. The first layer identifies situational contexts (e.g., "Communication with C2 using encoded subdomains"), while the second layer captures distinctive features that differentiate similar techniques (e.g., distinguishing T1132 "Data Encoding" from T1071 "Application Layer Protocol" based on whether the focus is on encoding methods or protocol usage). This structured approach provides a transparent basis for explainable TTP assignments and enhanced human oversight, while also helping to standardize other TTP extraction systems. Experiments show our framework (using Qwen2.5-32B) boosts Technique F1 scores by 11\% over GPT-4o. Qualitative analysis confirms superior standardization, enhanced transparency, and improved explainability in real-world threat intelligence scenarios. To the best of our knowledge, this is the first work that uses the LLM to generate, update, and apply the a new knowledge for TTP extraction.

Instantiating Standards: Enabling Standard-Driven Text TTP Extraction with Evolvable Memory

TL;DR

The paper tackles the challenge of extracting MITRE ATT&CK TTPs from unstructured CTI text while ensuring adherence to the official standard and providing transparency. It introduces a dual-layer Situational Knowledge Representation (SKR) paired with an Evolvable Memory System, both managed by an LLM, to generate, update, and apply contextual knowledge for TTP classification. A two-stage extraction pipeline (Stage1 retrieval/classification and Stage2 refinement/verification) enables accurate disambiguation and interoperability with other extraction systems, supported by memory generation, optimization, and forgetting mechanisms. Empirical results on the nguyen-etal-2024-noise dataset show substantial performance gains and improved standardization and explainability, with an higher F1 score than GPT-4o on procedures data, marking the first work to generate and apply new knowledge for TTP extraction using LLMs.

Abstract

Extracting MITRE ATT\&CK Tactics, Techniques, and Procedures (TTPs) from natural language threat reports is crucial yet challenging. Existing methods primarily focus on performance metrics using data-driven approaches, often neglecting mechanisms to ensure faithful adherence to the official standard. This deficiency compromises reliability and consistency of TTP assignments, creating intelligence silos and contradictory threat assessments across organizations. To address this, we introduce a novel framework that converts abstract standard definitions into actionable, contextualized knowledge. Our method utilizes Large Language Model (LLM) to generate, update, and apply this knowledge. This framework populates an evolvable memory with dual-layer situational knowledge instances derived from labeled examples and official definitions. The first layer identifies situational contexts (e.g., "Communication with C2 using encoded subdomains"), while the second layer captures distinctive features that differentiate similar techniques (e.g., distinguishing T1132 "Data Encoding" from T1071 "Application Layer Protocol" based on whether the focus is on encoding methods or protocol usage). This structured approach provides a transparent basis for explainable TTP assignments and enhanced human oversight, while also helping to standardize other TTP extraction systems. Experiments show our framework (using Qwen2.5-32B) boosts Technique F1 scores by 11\% over GPT-4o. Qualitative analysis confirms superior standardization, enhanced transparency, and improved explainability in real-world threat intelligence scenarios. To the best of our knowledge, this is the first work that uses the LLM to generate, update, and apply the a new knowledge for TTP extraction.
Paper Structure (15 sections, 2 figures, 2 tables)

This paper contains 15 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview of the Framework
  5. Situational Knowledge Representation (SKR)
  6. Life-long Memory Management
  7. Memory Generation
  8. Memory Optimization
  9. Memory Forget
  10. Standard-Driven TTP Extraction
  11. Stage 1: Initial Retrieval and Classification
  12. Stage 2: Refinement and Verification of TTP Classification
  13. Evaluation
  14. Limitations
  15. Conclusion

Figures (2)

  • Figure 1: Overview of the main components of the framework, including Situational Knowledge with memory system, knowledge generation and two-step knowledge-driven TTP extraction.
  • Figure 2: Key difference between the proposed Situational Knowledge Representation (SKR) and the other knowledge representation methods. Our SKR is a dual-layer knowledge representation, separate the retrieval text and classification guidance, thus alleviate the contradiction between the semantic similarity and contextual relevance.