CyberLLM-FINDS 2025: Instruction-Tuned Fine-tuning of Domain-Specific LLMs with Retrieval-Augmented Generation and Graph Integration for MITRE Evaluation

TL;DR

This work tackles the challenge of building domain-specific cybersecurity LLMs by fine-tuning Gemma-2B through instruction tuning on MITRE ATT&CK-aligned data and synthetic datasets generated with larger cloud LLMs. It integrates retrieval-augmented generation and a graph-based reasoning framework to improve multi-hop, long-context threat querying and TTP coverage, mapping outputs to structured ATT&CK labels. The study finds that small, locally runnable LLMs benefit from synthetic data and a hybrid training pipeline (including Update-qlora) but still face context-window limitations, which are mitigated by graph-augmented retrieval. Across RAG, graph, and graphRAG+GNN pipelines, the graph-enhanced retrieval approach demonstrates superior alignment to MITRE techniques while preserving efficiency, indicating practical potential for privacy-preserving cyber threat intelligence. The work advances scalable, interpretable cybersecurity LLMs capable of reasoning over structured threat intelligence in MITRE/ATT&CK contexts.

Abstract

Large Language Models (LLMs) such as Gemma-2B have shown strong performance in various natural language processing tasks. However, general-purpose models often lack the domain expertise required for cybersecurity applications. This work presents a methodology to fine-tune the Gemma-2B model into a domain-specific cybersecurity LLM. We detail the processes of dataset preparation, fine-tuning, and synthetic data generation, along with implications for real-world applications in threat detection, forensic investigation, and attack analysis. Experiments highlight challenges in prompt length distribution during domain-specific fine-tuning. Uneven prompt lengths limit the model's effective use of the context window, constraining local inference to 200-400 tokens despite hardware support for longer sequences. Chain-of-thought styled prompts, paired with quantized weights, yielded the best performance under these constraints. To address context limitations, we employed a hybrid strategy using cloud LLMs for synthetic data generation and local fine-tuning for deployment efficiency. To extend the evaluation, we introduce a Retrieval-Augmented Generation (RAG) pipeline and graph-based reasoning framework. This approach enables structured alignment with MITRE ATT&CK techniques through STIX-based threat intelligence, enhancing recall in multi-hop and long-context scenarios. Graph modules encode entity-neighborhood context and tactic chains, helping mitigate the constraints of short prompt windows. Results demonstrate improved model alignment with tactic, technique, and procedure (TTP) coverage, validating the utility of graph-augmented LLMs in cybersecurity threat intelligence applications.

Figures (7)

• Figure 1: Comparison of model accuracy across parameter sizes and prompting strategies as a function of the number of in-context examples. Prompted models consistently outperform non-prompted models, especially in few-shot regimes.
• Figure 2: Finetuning LLM
• Figure 3: Domain Finetuning LLM
• Figure 4: Workflow for generating synthetic cybersecurity data using large LLMs. The process produces instruction-format samples that encode attack behaviors, supporting Chain-of-Thought prompting and enabling small models to reason over structured threat intelligence such as MITRE’s Pyramid of Pain, which ranks the difficulty of detecting and disrupting various attacker artifacts.
• Figure 5: Instruction-tuning workflow using domain-adapted synthetic data. The pipeline integrates structured prompts derived from the MITRE ATT&CK framework and Chain-of-Thought prompting strategies. The model is fine-tuned with a mixture of zero-shot, one-shot, and few-shot examples to support reasoning and classification across cybersecurity tasks.