Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Labeling NIDS Rules with MITRE ATT&CK Techniques: Machine Learning vs. Large Language Models

Nir Daniel, Florian Klaus Kaiser, Shay Giladi, Sapir Sharabi, Raz Moyal, Shalev Shpolyansky, Andres Murillo, Aviad Elyashar, Rami Puzis

TL;DR

This work benchmarks labeling NIDS rules with MITRE ATT&CK techniques by comparing three prominent LLMs (ChatGPT, Claude, Gemini) against ML models trained on TF-IDF features, using a dataset of 973 Snort rules mapped to 75 ATT&CK techniques. It formalizes the task as conditional text generation for LLMs and a supervised multi-label classification pipeline for ML, with a balanced 80/20 train/test split. Across configurations, ML models achieve higher precision, recall, and F1 than LLMs (e.g., technique F1 up to 0.87 and tactic F1 up to 0.92), while LLMs provide explainable mappings and flexible reasoning, especially when prompt templates combine contextual and example-driven guidance (T-$ICL_2$). The study also delivers a resource: a labeled NIDS rule dataset for future benchmarking, and argues for hybrid LLM-ML approaches to harness explainability and high accuracy for SOC workflows and evolving threat landscapes.

Abstract

Analysts in Security Operations Centers (SOCs) are often occupied with time-consuming investigations of alerts from Network Intrusion Detection Systems (NIDS). Many NIDS rules lack clear explanations and associations with attack techniques, complicating the alert triage and the generation of attack hypotheses. Large Language Models (LLMs) may be a promising technology to reduce the alert explainability gap by associating rules with attack techniques. In this paper, we investigate the ability of three prominent LLMs (ChatGPT, Claude, and Gemini) to reason about NIDS rules while labeling them with MITRE ATT&CK tactics and techniques. We discuss prompt design and present experiments performed with 973 Snort rules. Our results indicate that while LLMs provide explainable, scalable, and efficient initial mappings, traditional Machine Learning (ML) models consistently outperform them in accuracy, achieving higher precision, recall, and F1-scores. These results highlight the potential for hybrid LLM-ML approaches to enhance SOC operations and better address the evolving threat landscape.

Labeling NIDS Rules with MITRE ATT&CK Techniques: Machine Learning vs. Large Language Models

TL;DR

This work benchmarks labeling NIDS rules with MITRE ATT&CK techniques by comparing three prominent LLMs (ChatGPT, Claude, Gemini) against ML models trained on TF-IDF features, using a dataset of 973 Snort rules mapped to 75 ATT&CK techniques. It formalizes the task as conditional text generation for LLMs and a supervised multi-label classification pipeline for ML, with a balanced 80/20 train/test split. Across configurations, ML models achieve higher precision, recall, and F1 than LLMs (e.g., technique F1 up to 0.87 and tactic F1 up to 0.92), while LLMs provide explainable mappings and flexible reasoning, especially when prompt templates combine contextual and example-driven guidance (T-). The study also delivers a resource: a labeled NIDS rule dataset for future benchmarking, and argues for hybrid LLM-ML approaches to harness explainability and high accuracy for SOC workflows and evolving threat landscapes.

Abstract

Analysts in Security Operations Centers (SOCs) are often occupied with time-consuming investigations of alerts from Network Intrusion Detection Systems (NIDS). Many NIDS rules lack clear explanations and associations with attack techniques, complicating the alert triage and the generation of attack hypotheses. Large Language Models (LLMs) may be a promising technology to reduce the alert explainability gap by associating rules with attack techniques. In this paper, we investigate the ability of three prominent LLMs (ChatGPT, Claude, and Gemini) to reason about NIDS rules while labeling them with MITRE ATT&CK tactics and techniques. We discuss prompt design and present experiments performed with 973 Snort rules. Our results indicate that while LLMs provide explainable, scalable, and efficient initial mappings, traditional Machine Learning (ML) models consistently outperform them in accuracy, achieving higher precision, recall, and F1-scores. These results highlight the potential for hybrid LLM-ML approaches to enhance SOC operations and better address the evolving threat landscape.

Paper Structure

This paper contains 28 sections, 6 equations, 5 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Cyber Threat Intelligence
  4. Intrusion Detection Systems
  5. Large Language Models
  6. Related Work
  7. Labeling reports with CTI
  8. Labeling of host-related data with CTI
  9. Labeling of network-related data with CTI
  10. Labeling Network Packets
  11. Network Intrusion Detection Rules Labeling
  12. Materials and Methods
  13. Prompt Engineering for the NIDS Labeling Task
  14. Problem Definition
  15. Prompt Template Generation
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Snort rule example
  • Figure 2: Materials and methods overview
  • Figure 3: Distribution rules across ATT&CK techniques for both the train and the test sets
  • Figure 4: Distribution rules across ATT&CK tactics for both the train and the test sets
  • Figure 5: Precision and recall of LLM-based technique labeling across different configurations