Deep Active Learning for Data Mining from Conflict Text Corpora

TL;DR

The paper presents an active-learning framework to mine micro-dynamics from conflict text corpora, using an unsupervised seed step to bootstrap a supervised encoder-based LLM (e.g., ConfliBERT or DistilBERT) for extracting sub-classes of conflict events with minimal human annotation. By combining a two-stage seed-filtering process, a batch-wise querying strategy, and Monte Carlo convergence checks, the method achieves performance close to gold-standard labeling while reducing annotation effort by up to 99%. Empirical validation on two gold-standard datasets (religious violence and electoral violence) demonstrates strong AP and AUROC performance, with convergence typically around round 6–7 and robustness across batch sizes and learning rates. The work highlights the practical potential for scalable, cost-effective extraction of rich, event-level features from large text corpora, while acknowledging limitations related to extreme class imbalance and domain-specific generalization.

Abstract

High-resolution event data on armed conflict and related processes have revolutionized the study of political contention with datasets like UCDP GED, ACLED etc. However, most of these datasets limit themselves to collecting spatio-temporal (high-resolution) and intensity data. Information on dynamics, such as targets, tactics, purposes etc. are rarely collected owing to the extreme workload of collecting data. However, most datasets rely on a rich corpus of textual data allowing further mining of further information connected to each event. This paper proposes one such approach that is inexpensive and high performance, leveraging active learning - an iterative process of improving a machine learning model based on sequential (guided) human input. Active learning is employed to then step-wise train (fine-tuning) of a large, encoder-only language model adapted for extracting sub-classes of events relating to conflict dynamics. The approach shows performance similar to human (gold-standard) coding while reducing the amount of required human annotation by as much as 99%.

Figures (6)

• Figure 1: A schematic representation of the active learning approach employed in the paper. The top row of the figure describes the fully unsupervised step (the extraction of the "core" seed dataset for annotation). Below, the active learning iterative loop is presented. At this step, a model $M$ is iteratively improved through sequential training. This is done using batches of $k$ human annotated observations (news articles) that are selected from the core dataset for annotation by the previous iteration of $M$.
• Figure 2: Receiver-Operator Curve (ROC) and Area Under the Precision-Recall Curve (AUPR), including bootstrapped confidence intervals for the purely unsupervised approach for the two datasets - top (blue) for religious violence, bottom (gray) for political (electoral) violence.
• Figure 3: Cross-validation experiment (with complete data) between three classes of natural-language classification models (8 models) for extracting religious violence events. Left pane - ROC curves (note high imbalance), Right pane - Precision Recall Curve. In both cases higher curves indicate better performance. Blue models are classical (shallow learning) approaches; dashed brown lines are decoder-only LLMs; solid brown lines are encoder-only LLMs.
• Figure 4: Average Precision (AP) and Area Under the Receiver Operator Curves (AUROC) across active learning rounds for three batch sizes (50, 100 and 200 articles) on the two complete datasets: religious and electoral violence. Learning rate for all experiments : $5 \times 10^{-5}$
• Figure 5: Impact of learning rates on the performance of the ConfliBERT-derived classifier predicting religious violence.