Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal

Anastasiia Sedova, Lena Zellinger, Benjamin Roth

TL;DR

AGRA addresses label noise by replacing pre-cleaning with adaptive, gradient-based outlier removal during training. It computes per-sample gradients $\\nabla\\widetilde{L}(x_t,y_t)$ and a batch comparison gradient $\\nabla\\widetilde{L}_{com}$, using their cosine similarity $sim_{y_t}$ to decide whether to keep, remove, or relabel samples via flexible rules such as $sim_{y_t} \le 0$ or $sim_{y'} > sim_{y_t}$. The method is compatible with standard losses and a differentiable $F_1$ loss, and it employs class-aware sampling for the comparison batch to handle imbalance. Across seven data-rich experiments spanning NLP and vision, AGRA outperforms several baselines, sometimes matching a model trained on gold labels, and ablations show the benefit of $F_1$-based losses and weighted sampling in noisy settings. This work demonstrates that dynamic, gradient-guided data usage during training can yield robust performance on noisy data without assuming a particular noise distribution, with practical implications for weak supervision and large-scale labeling regimes.

Abstract

An accurate and substantial dataset is essential for training a reliable and well-performing model. However, even manually annotated datasets contain label errors, not to mention automatically labeled ones. Previous methods for label denoising have primarily focused on detecting outliers and their permanent removal - a process that is likely to over- or underfilter the dataset. In this work, we propose AGRA: a new method for learning with noisy labels by using Adaptive GRAdient-based outlier removal. Instead of cleaning the dataset prior to model training, the dataset is dynamically adjusted during the training process. By comparing the aggregated gradient of a batch of samples and an individual example gradient, our method dynamically decides whether a corresponding example is helpful for the model at this point or is counter-productive and should be left out for the current update. Extensive evaluation on several datasets demonstrates AGRA's effectiveness, while a comprehensive results analysis supports our initial hypothesis: permanent hard outlier removal is not always what model benefits the most from.

Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal

TL;DR

AGRA addresses label noise by replacing pre-cleaning with adaptive, gradient-based outlier removal during training. It computes per-sample gradients and a batch comparison gradient , using their cosine similarity to decide whether to keep, remove, or relabel samples via flexible rules such as or . The method is compatible with standard losses and a differentiable loss, and it employs class-aware sampling for the comparison batch to handle imbalance. Across seven data-rich experiments spanning NLP and vision, AGRA outperforms several baselines, sometimes matching a model trained on gold labels, and ablations show the benefit of -based losses and weighted sampling in noisy settings. This work demonstrates that dynamic, gradient-guided data usage during training can yield robust performance on noisy data without assuming a particular noise distribution, with practical implications for weak supervision and large-scale labeling regimes.

Abstract

An accurate and substantial dataset is essential for training a reliable and well-performing model. However, even manually annotated datasets contain label errors, not to mention automatically labeled ones. Previous methods for label denoising have primarily focused on detecting outliers and their permanent removal - a process that is likely to over- or underfilter the dataset. In this work, we propose AGRA: a new method for learning with noisy labels by using Adaptive GRAdient-based outlier removal. Instead of cleaning the dataset prior to model training, the dataset is dynamically adjusted during the training process. By comparing the aggregated gradient of a batch of samples and an individual example gradient, our method dynamically decides whether a corresponding example is helpful for the model at this point or is counter-productive and should be left out for the current update. Extensive evaluation on several datasets demonstrates AGRA's effectiveness, while a comprehensive results analysis supports our initial hypothesis: permanent hard outlier removal is not always what model benefits the most from.
Paper Structure (28 sections, 9 equations, 2 figures, 6 tables, 2 algorithms)

This paper contains 28 sections, 9 equations, 2 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning with Noisy Labels.
  4. Outlier Detection.
  5. Weak Supervision.
  6. Gradient-based Approaches.
  7. AGRA: Adaptive Gradient-Based Outlier Removal
  8. Notation
  9. Algorithm Description
  10. Comparison Batch Sampling
  11. Selection of Comparison Loss Functions
  12. Experiments
  13. Datasets
  14. Baselines
  15. Experimental Setup
  16. ...and 13 more sections

Figures (2)

  • Figure 1: AGRA method for learning with noisy data. Each sample in the update batch is decided to be either kept for further model training or removed depending on the similarity of its gradient to the aggregated gradient of the comparison batch sampled from the same data.
  • Figure 2: Case study on the YouTube dataset. The plots represent the percentage of samples in each batch that were correctly kept, correctly removed, falsely kept and falsely removed during the training of the best-performing models for all combinations of comparison losses and sampling strategies.