STENCIL: Submodular Mutual Information Based Weak Supervision for Cold-Start Active Learning

TL;DR

STENCIL tackles annotation cost in fine-tuning NLP models under class imbalance and cold-start by using a compact exemplar set of rare-class instances to steer active learning. It frames selection as maximizing Submodular Mutual Information between a candidate unlabeled set and the exemplar query set, employing several submodular instantiations (e.g., LOGDETMI, GCMI) with GloVe-based featurization. A single greedy round selects a batch of unlabeled instances to weakly label and then strongly label for fine-tuning, achieving 10–18% gains in overall accuracy and 17–40% gains in rare-class F1 across YouTube, SMS, and Twitter datasets. The work provides a practical, plug-in exemplar-guided approach for fast improvements in cold-start active learning for NLP and highlights the value of diversity and exemplar relevance in SMI-based selection.

Abstract

As supervised fine-tuning of pre-trained models within NLP applications increases in popularity, larger corpora of annotated data are required, especially with increasing parameter counts in large language models. Active learning, which attempts to mine and annotate unlabeled instances to improve model performance maximally fast, is a common choice for reducing the annotation cost; however, most methods typically ignore class imbalance and either assume access to initial annotated data or require multiple rounds of active learning selection before improving rare classes. We present STENCIL, which utilizes a set of text exemplars and the recently proposed submodular mutual information to select a set of weakly labeled rare-class instances that are then strongly labeled by an annotator. We show that STENCIL improves overall accuracy by $10\%-18\%$ and rare-class F-1 score by $17\%-40\%$ on multiple text classification datasets over common active learning methods within the class-imbalanced cold-start setting.

Figures (1)

• Figure 1: Architectural process of STENCIL. A downstream task is presented, wherein subsequent data collection transpires. An expert generates an exemplar query set based on prior knowledge of the task. These artifacts are fed to STENCIL's selection process, and the resulting selected subset is annotated and used for fine-tuning.