Revealing the True Indicators: Understanding and Improving IoC Extraction From Threat Reports

TL;DR

The paper tackles the problem of reliably extracting IoCs from unstructured threat reports, where ground-truth quality critically affects evaluation and downstream use. It introduces LANCE, a hybrid system that combines RegEx-based extraction with explainable, LLM-driven labeling and a human-in-the-loop validation workflow, operationalized through a custom UI. This approach enables the creation of PRISM, a publicly available, expert-validated IoC benchmark that supports training and fair evaluation of IoC extraction methods. Empirical results show that LANCE outperforms traditional GT creation methods, generalizes across multiple LLMs, and reduces analyst workload by $43\%$, demonstrating the practical value of a structured, explainable HITL pipeline for cybersecurity threat intelligence. Overall, the work provides a reproducible framework and dataset that can drive more trustworthy and scalable IoC extraction in real-world settings.

Abstract

Indicators of Compromise (IoCs) are critical for threat detection and response, marking malicious activity across networks and systems. Yet, the effectiveness of automated IoC extraction systems is fundamentally limited by one key issue: the lack of high-quality ground truth. Current extraction tools rely either on manually extracted ground truth, which is labor-intensive and costly, or on automated ground truth creation methods that include non-malicious artifacts, leading to inflated false positive (FP) rates and unreliable threat intelligence. In this work, we analyze the shortcomings of existing ground truth creation strategies and address them by introducing the first hybrid human-in-the-loop pipeline for IoC extraction, which combines a large language model-based classifier (LANCE) with expert analyst validation. Our system improves precision through explainable, context-aware labeling and reduces analysts' work factor by 43% compared to manual annotation, as demonstrated in our evaluation with six analysts. Using this approach, we produce PRISM, a high-quality, publicly available benchmark of 1,791 labeled IoCs from 50 real-world threat reports. PRISM supports both fair evaluation and training of IoC extraction methods and enables reproducible research grounded in expert-validated indicators.

Figures (13)

• Figure 1: Overview of the human-in-the-loop (HITL) IoC extraction pipeline. It combines automated extraction and LLM-based labeling (LANCE) with manual annotation, thus promoting efficient, high-confidence IoC labeling.
• Figure 2: Overview of the LANCE pipeline. Indicators are extracted using regular expressions, labeled by an LLM using contextual report segments, and finalized through a voting mechanism to resolve overlapping predictions.
• Figure 3: Structure of the manual annotation process used to create the PRISM dataset. The process includes two phases: the Baseline Annotation Pass (BAP), where analysts label indicators without assistance, and the Guided Annotation Pass (GAP), where analysts review indicators pre-labeled by LANCE along with justifications. In both phases, labels from junior analysts are compared, and any disagreements are resolved by a senior analyst. The final consensus labels are incorporated into the PRISM dataset.
• Figure 4: Agreement ratios across IoC types during the manual annotation process. The left plot shows results from BAP (Baseline Annotation Pass) and the right plot from GAP (Guided Annotation Pass). Each bar pair reflects agreement between the junior analysts (Analysts Only) and including LANCE (All).
• Figure 5: Comparison of LANCE and analyst performance across precision, recall, and F1 score during both annotation phases. The top row shows results from BAP, while the bottom row shows GAP.