Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimation

Jianan Chen, Vishwesh Ramanathan, Tony Xu, Anne L. Martel

TL;DR

The paper addresses label noise in supervised learning, particularly in medical settings, by observing that cross-validation fluctuations across folds can reflect noisy labels and thus be informative for detection. It introduces ReCoV, a model-agnostic, parameter-free detector, and fastReCoV, a scalable variant, that identify noisy samples by tracking which samples repeatedly appear in the worst-performing folds during repeated $k$-fold cross-validation and related Monte-Carlo analyses. The methods are validated across binary (Mushroom), multi-class (CIFAR-10N), survival (HECKTOR), and regression (PANDA) tasks, achieving state-of-the-art noise-detection performance and improving retrained model accuracy, including survival-model concordance-index improvements. The work provides practical, plug-in tools for data cleaning in diverse domains and suggests potential applicability to medical foundation-model workflows.

Abstract

Machine learning models experience deteriorated performance when trained in the presence of noisy labels. This is particularly problematic for medical tasks, such as survival prediction, which typically face high label noise complexity with few clear-cut solutions. Inspired by the large fluctuations across folds in the cross-validation performance of survival analyses, we design Monte-Carlo experiments to show that such fluctuation could be caused by label noise. We propose two novel and straightforward label noise detection algorithms that effectively identify noisy examples by pinpointing the samples that more frequently contribute to inferior cross-validation results. We first introduce Repeated Cross-Validation (ReCoV), a parameter-free label noise detection algorithm that is robust to model choice. We further develop fastReCoV, a less robust but more tractable and efficient variant of ReCoV suitable for deep learning applications. Through extensive experiments, we show that ReCoV and fastReCoV achieve state-of-the-art label noise detection performance in a wide range of modalities, models and tasks, including survival analysis, which has yet to be addressed in the literature. Our code and data are publicly available at https://github.com/GJiananChen/ReCoV.

Cross-Validation Is All You Need: A Statistical Approach To Label Noise Estimation

TL;DR

The paper addresses label noise in supervised learning, particularly in medical settings, by observing that cross-validation fluctuations across folds can reflect noisy labels and thus be informative for detection. It introduces ReCoV, a model-agnostic, parameter-free detector, and fastReCoV, a scalable variant, that identify noisy samples by tracking which samples repeatedly appear in the worst-performing folds during repeated -fold cross-validation and related Monte-Carlo analyses. The methods are validated across binary (Mushroom), multi-class (CIFAR-10N), survival (HECKTOR), and regression (PANDA) tasks, achieving state-of-the-art noise-detection performance and improving retrained model accuracy, including survival-model concordance-index improvements. The work provides practical, plug-in tools for data cleaning in diverse domains and suggests potential applicability to medical foundation-model workflows.

Abstract

Machine learning models experience deteriorated performance when trained in the presence of noisy labels. This is particularly problematic for medical tasks, such as survival prediction, which typically face high label noise complexity with few clear-cut solutions. Inspired by the large fluctuations across folds in the cross-validation performance of survival analyses, we design Monte-Carlo experiments to show that such fluctuation could be caused by label noise. We propose two novel and straightforward label noise detection algorithms that effectively identify noisy examples by pinpointing the samples that more frequently contribute to inferior cross-validation results. We first introduce Repeated Cross-Validation (ReCoV), a parameter-free label noise detection algorithm that is robust to model choice. We further develop fastReCoV, a less robust but more tractable and efficient variant of ReCoV suitable for deep learning applications. Through extensive experiments, we show that ReCoV and fastReCoV achieve state-of-the-art label noise detection performance in a wide range of modalities, models and tasks, including survival analysis, which has yet to be addressed in the literature. Our code and data are publicly available at https://github.com/GJiananChen/ReCoV.
Paper Structure (15 sections, 3 equations, 2 figures, 5 tables, 2 algorithms)

This paper contains 15 sections, 3 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methods
  3. Assumptions
  4. Modeling cross-validations in classification task
  5. Repeated Cross-Validations (ReCoV) for noise detection
  6. FastReCoV
  7. Experiments
  8. Binary classification on Mushroom with ReCoV
  9. Multi-class classification on CIFAR-10N with fastReCoV
  10. Survival prediction on HECKTOR with ReCoV and fastReCoV
  11. Regression on PANDA with fastReCoV
  12. Results
  13. Conclusions and Discussion
  14. Acknowledgments.
  15. Disclosure of Interests.

Figures (2)

  • Figure 1: Validation performance (concordance-index) on the TCIA Head-Neck-PET-CT dataset, comparing performance pre- and post-cleaning of noisy samples.
  • Figure 2: Results on the Mushroom dataset (N=8124, noise ratio=10%) matches Monte Carlo simulations. Number of runs are selected to show 4.5%, 0.3% and $\sim$0 overlap of noisy and clean sample distribution. Dashed lines in the Monte Carlo plots refer to the theoretical separation thresholds.