Benchmarking noisy label detection methods

TL;DR

The paper tackles the problem of detecting noisy labels to enable cleaner training and evaluation. It introduces a unified benchmark that decomposes detection methods into a label agreement function and an aggregation strategy, comparing in-sample versus out-of-sample information gathering under a fixed noise-rate budget $\eta$. Through extensive experiments on vision and tabular datasets with both synthetic and real-world noise, the study finds that aggregating prediction probabilities using Mean Probabilities together with the Logit Margin as the label agreement function (Mean Prob - LM) delivers the most robust performance across diverse scenarios. The results challenge the necessity of complex detection schemes based on training dynamics, showing that simpler in-sample methods often outperform their out-of-sample counterparts, with practical implications for dataset cleaning and reliable evaluation.

Abstract

Label noise is a common problem in real-world datasets, affecting both model training and validation. Clean data are essential for achieving strong performance and ensuring reliable evaluation. While various techniques have been proposed to detect noisy labels, there is no clear consensus on optimal approaches. We perform a comprehensive benchmark of detection methods by decomposing them into three fundamental components: label agreement function, aggregation method, and information gathering approach (in-sample vs out-of-sample). This decomposition can be applied to many existing detection methods, and enables systematic comparison across diverse approaches. To fairly compare methods, we propose a unified benchmark task, detecting a fraction of training samples equal to the dataset's noise rate. We also introduce a novel metric: the false negative rate at this fixed operating point. Our evaluation spans vision and tabular datasets under both synthetic and real-world noise conditions. We identify that in-sample information gathering using average probability aggregation combined with the logit margin as the label agreement function achieves the best results across most scenarios. Our findings provide practical guidance for designing new detection methods and selecting techniques for specific applications.

Figures (10)

• Figure 1: Illustration of the dataset cleaning process: Starting from a noisy dataset, scores are calculated to identify potentially mislabeled samples. Samples are detected as noisy or clean based on a threshold, with noisy samples relabeled to produce a cleaned dataset for downstream tasks.
• Figure 2: Data gathering representation for in-sample and out-of-sample methods. Blue circles indicate samples for which predictions are stored, red circles indicate samples used only for training and purple samples indicate samples in which both training and gathering of predictions are made.
• Figure 3: Example of the budget x FNR curve for an ideal, random and a real method in a setting with $\eta = 0.2$.
• Figure 4: Workflow for training and post-hoc detection optimization. The model is trained once, and the logits for each sample are saved across epochs. A small detection tuning set is used to select the best aggregation window or CTRL hyperparameters, which are then evaluated in the remaining dataset.
• Figure 5: The dataset splits used in this work. The dataset is divided into the usual train-validation-test splits for classification. One additional type of split is made for detection, represented by the three tuning splits used for tuning the detection methods.