From Retrieval to Reasoning: A Framework for Cyber Threat Intelligence NER with Explicit and Adaptive Instructions

TL;DR

CTI NER faces limitations with retrieval-based in-context learning, where success is driven more by incidental entity-type overlaps than by genuine semantic understanding. The authors introduce TTPrompt, a hierarchical explicit-instruction framework that maps CTI analysis into Task-Definition, Guiding-Strategy, and Annotation-Guideline components, complemented by Feedback-driven Instruction Refinement (FIR) that self-adjusts guidelines using minimal labeled data. Across five CTI NER benchmarks, TTPrompt consistently outperforms RetICL baselines and rivals, or matches, fully fine-tuned models, especially on harder datasets, demonstrating strong data efficiency and adaptability. The work highlights the insufficiency of implicit induction in NER and provides a practical pathway to robust, dataset-adaptive NER in rapidly evolving cyber-threat landscapes.

Abstract

The automation of Cyber Threat Intelligence (CTI) relies heavily on Named Entity Recognition (NER) to extract critical entities from unstructured text. Currently, Large Language Models (LLMs) primarily address this task through retrieval-based In-Context Learning (ICL). This paper analyzes this mainstream paradigm, revealing a fundamental flaw: its success stems not from global semantic similarity but largely from the incidental overlap of entity types within retrieved examples. This exposes the limitations of relying on unreliable implicit induction. To address this, we propose TTPrompt, a framework shifting from implicit induction to explicit instruction. TTPrompt maps the core concepts of CTI's Tactics, Techniques, and Procedures (TTPs) into an instruction hierarchy: formulating task definitions as Tactics, guiding strategies as Techniques, and annotation guidelines as Procedures. Furthermore, to handle the adaptability challenge of static guidelines, we introduce Feedback-driven Instruction Refinement (FIR). FIR enables LLMs to self-refine guidelines by learning from errors on minimal labeled data, adapting to distinct annotation dialects. Experiments on five CTI NER benchmarks demonstrate that TTPrompt consistently surpasses retrieval-based baselines. Notably, with refinement on just 1% of training data, it rivals models fine-tuned on the full dataset. For instance, on LADDER, its Micro F1 of 71.96% approaches the fine-tuned baseline, and on the complex CTINexus, its Macro F1 exceeds the fine-tuned ACLM model by 10.91%.

Figures (5)

• Figure 1: Empirical analysis of the RetICL mechanism. (a) Baseline performance of the Semantic-kNN method across five CTI NER benchmarks. (b) Impact of entity type overlap on extraction performance. The significant gap between "Type OVERLAP" and "Type NO OVERLAP" reveals the model's heavy reliance on incidental type matching. Note that CTINexus is omitted here as all its cases fall under Type OVERLAP. (c) Performance comparison of three distinct retrieval paradigms. The superiority of the Type-Overlap paradigm over Semantic-kNN and Entity-Density further verifies that explicit type guidance is more effective than implicit semantic induction.
• Figure 2: This is a system overview of our TTPrompt framework. The process involves generating initial annotation Guidelines, selecting the optimal strategy, and iteratively refining these guidelines through the Feedback-driven Instruction Refinement mechanism. The Reflection Agent and Editor Agent are used to implement the core refinement loop; they calculate a semantic gradient from prediction errors to progressively optimize the final instruction set used for inference.
• Figure 3: Performance comparison of strategies generated by three different LLMs.
• Figure 4: Performance Gain ($\Delta F1$) vs. Dataset Difficulty ($\Omega$).
• Figure 5: Performance of the FIR process over 10 epochs.