Next Generation Active Learning: Mixture of LLMs in the Loop

TL;DR

This work addresses the high cost and variable quality of human labeling in active learning by introducing MoLLIA, a framework that replaces human annotators with a Mixture-of-LLMs-based annotator (MoLAM). MoLAM aggregates outputs from multiple lightweight LLMs and employs semi-supervised pseudo-labeling to produce high-quality labels, while robust active learning employs annotation discrepancy and negative learning to mitigate label noise. The approach is validated on four text classification benchmarks, demonstrating annotation performance near or surpassing human annotators and strong gains over single-LLM and some ensemble baselines, all while running on local hardware. The results suggest a practical, privacy-friendly path toward fully automated, robust active learning in real-world settings where data-labeling costs are critical.

Abstract

With the rapid advancement and strong generalization capabilities of large language models (LLMs), they have been increasingly incorporated into the active learning pipelines as annotators to reduce annotation costs. However, considering the annotation quality, labels generated by LLMs often fall short of real-world applicability. To address this, we propose a novel active learning framework, Mixture of LLMs in the Loop Active Learning, replacing human annotators with labels generated through a Mixture-of-LLMs-based annotation model, aimed at enhancing LLM-based annotation robustness by aggregating the strengths of multiple LLMs. To further mitigate the impact of the noisy labels, we introduce annotation discrepancy and negative learning to identify the unreliable annotations and enhance learning effectiveness. Extensive experiments demonstrate that our framework achieves performance comparable to human annotation and consistently outperforms single-LLM baselines and other LLM-ensemble-based approaches. Moreover, our framework is built on lightweight LLMs, enabling it to operate fully on local machines in real-world applications.

Figures (12)

• Figure 1: Overview of MoLLIA.
• Figure 2: Workflow of MoLLIA framework. The AL model is first trained on the initial labeled dataset and used to query the most informative instances for annotation. A Mixture-of-LLMs-based annotator then generates labels $y^+$ and corresponding negative labels $y^-$ for the selected instances (as detailed in Mixture-of-LLMs-based Annotation Model section). The annotation discrepancy ($d_{anno}$) is computed based on the disagreement between the AL model’s predictions and the Mixture-of-LLMs-based annotator. The AL model is then updated using a loss function that incorporates both weighted annotation discrepancy and negative learning, and the querying process is iteratively repeated based on the updated AL model.
• Figure 3: Overview of MoLAM.
• Figure 4: Averaged micro-F1 score with BEMPS on DistilBERT, averaged results with 5 random seeds.
• Figure 5: Averaged micro-F1 score with NoiseStability on DistilBERT, averaged results with 5 random seeds.