NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification

TL;DR

NoisyAG-News introduces a human-annotated benchmark for instance-dependent label noise in text classification, addressing the gap between synthetic noise and real-world noise patterns. The paper analyzes noisy transition patterns and demonstrates that human noise is feature-dependent and significantly different from synthetic noise, challenging current PLMs and LNL methods. Through extensive experiments with multiple transformer models and noise-handling techniques, the authors show that instance-dependent noise degrades performance more than class-conditional noise and that standard LNL approaches often fail, highlighting the need for IDN-aware methods. The benchmark provides a controlled, scalable resource to evaluate and develop robust text classification methods under realistic noise conditions and can guide future research in learning with noisy labels for NLP tasks.

Abstract

Existing research on learning with noisy labels predominantly focuses on synthetic label noise. Although synthetic noise possesses well-defined structural properties, it often fails to accurately replicate real-world noise patterns. In recent years, there has been a concerted effort to construct generalizable and controllable instance-dependent noise datasets for image classification, significantly advancing the development of noise-robust learning in this area. However, studies on noisy label learning for text classification remain scarce. To better understand label noise in real-world text classification settings, we constructed the benchmark dataset NoisyAG-News through manual annotation. Initially, we analyzed the annotated data to gather observations about real-world noise. We qualitatively and quantitatively demonstrated that real-world noisy labels adhere to instance-dependent patterns. Subsequently, we conducted comprehensive learning experiments on NoisyAG-News and its corresponding synthetic noise datasets using pre-trained language models and noise-handling techniques. Our findings reveal that while pre-trained models are resilient to synthetic noise, they struggle against instance-dependent noise, with samples of varying confusion levels showing inconsistent performance during training and testing. These real-world noise patterns pose new, significant challenges, prompting a reevaluation of noisy label handling methods. We hope that NoisyAG-News will facilitate the development and evaluation of future solutions for learning with noisy labels.

Figures (13)

• Figure 1: NTMs for Three Annotation Group, NoisyAg-NewsMed and Corresponding Synthetic Noise.
• Figure 2: ${p}_{i, \nu}[j]$ in NoisyAG-NewsMed and Synthetic Noise for Class World and Sci/Tech.
• Figure 3: DataSet Decomposition, State Definition and Transition.
• Figure 4: Acc. on NoisyAG-NewsMed and NoisyAG-NewsMed-NTM.
• Figure 5: Confusion of Different Group.