Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance

TL;DR

This work tackles the challenge of learning from noisy labels in text classification by introducing NoiseAL, a collaborative active-learning framework that couples two small models (SMs) with an LLM-based annotator. A co-prediction network of SMs, guided by a dynamic-enhanced threshold, partitions data into Consistent/Discrepant sets and further into R (clean), P (purified), and H (hard) subsets; the LLM is then used to generate labels for P and provide demonstrations from R to bolster in-context learning. The SMs are trained with targeted losses on each subset—cross-entropy on R, reversed cross-entropy on P, and EmbMix-based regularization on H—resulting in a final objective L = L_R + L_P + L_H. Extensive experiments on synthetic and real-world noisy datasets show NoiseAL consistently outperforms state-of-the-art baselines, demonstrates robustness to instance-dependent noise, and highlights cost-effective use of LLMs for label denoising. The approach offers a practical pathway to scalable, noise-robust learning by fusing SM filters with LLM-powered correction and demonstration-enabled ICL.

Abstract

Learning from noisy labels (LNL) is a challenge that arises in many real-world scenarios where collected training data can contain incorrect or corrupted labels. Most existing solutions identify noisy labels and adopt active learning to query human experts on them for denoising. In the era of large language models (LLMs), although we can reduce the human effort to improve these methods, their performances are still subject to accurately separating the clean and noisy samples from noisy data. In this paper, we propose an innovative collaborative learning framework NoiseAL based on active learning to combine LLMs and small models (SMs) for learning from noisy labels. During collaborative training, we first adopt two SMs to form a co-prediction network and propose a dynamic-enhanced threshold strategy to divide the noisy data into different subsets, then select the clean and noisy samples from these subsets to feed the active annotator LLMs to rectify noisy samples. Finally, we employ different optimization objectives to conquer subsets with different degrees of label noises. Extensive experiments on synthetic and real-world noise datasets further demonstrate the superiority of our framework over state-of-the-art baselines.

Figures (9)

• Figure 1: Comparisons of our proposed NoiseAL with previous LNL methods on the Trec dataset under different noise scenarios. NoiseAL surpasses all other baselines and under some scenarios near the performance supervised on ground truth labels.
• Figure 2: The loss distributions of Bert on Trec dataset under 40% asymmetric noise in different training stages. The solid line represents the loss distributions, and the dashed line points out the mean value of loss distributions. During training, the clean samples tend to have a smaller loss value and the noisy samples tend to have a bigger loss value. And the loss distributions of clean and noisy samples are becoming more consistent.
• Figure 3: The overview of NoiseAL. During collaborative training, the SMs serve as a filter: (1) employs a co-prediction network (strong model and weak model) to obtain different confidences ($p_s$ and $p_w$). Based on $p_s$ and $p_w$, (2) the dynamic-enhanced selection module first divides the noisy data into consistency set $\mathcal{C}$ and discrepancy set $\mathcal{I}$, then groups these two sets into the clean set $\mathcal{R}$, hard set $\mathcal{H}$, and purified set $\mathcal{P}$. Meanwhile, the LLMs serve as active annotators: (1) construct demonstrations by selecting clean samples from $\mathcal{R}$, which can prompt its ICL performance; (2) query the noisy sample from $\mathcal{P}$ and generate labels to imbue its knowledge to SMs.
• Figure 4: The confidence distributions (a-c, e-g) and loss distributions (d,h) of Base/Ours on Trec under 40% asymmetric noise in different stages. We observe that the base model (a-c) gradually overfits the noisy sample, while our method (e-g) keeps learning from clean samples effectively and eventually avoids fitting noisy samples.
• Figure 5: Comparisions of NoiseAL with KNN on Trec dataset under 20% and 40% asymmetric label noise.