Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

L2B: Learning to Bootstrap Robust Models for Combating Label Noise

Yuyin Zhou, Xianhang Li, Fengze Liu, Qingyue Wei, Xuxi Chen, Lequan Yu, Cihang Xie, Matthew P. Lungren, Lei Xing

TL;DR

A simple and effective method, named Learning to Bootstrap (L2B), which enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels, by dynamically adjusting the importance weight between real observed and generated labels.

Abstract

Deep neural networks have shown great success in representation learning. However, when learning with noisy labels (LNL), they can easily overfit and fail to generalize to new data. This paper introduces a simple and effective method, named Learning to Bootstrap (L2B), which enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels. It achieves this by dynamically adjusting the importance weight between real observed and generated labels, as well as between different samples through meta-learning. Unlike existing instance reweighting methods, the key to our method lies in a new, versatile objective that enables implicit relabeling concurrently, leading to significant improvements without incurring additional costs. L2B offers several benefits over the baseline methods. It yields more robust models that are less susceptible to the impact of noisy labels by guiding the bootstrapping procedure more effectively. It better exploits the valuable information contained in corrupted instances by adapting the weights of both instances and labels. Furthermore, L2B is compatible with existing LNL methods and delivers competitive results spanning natural and medical imaging tasks including classification and segmentation under both synthetic and real-world noise. Extensive experiments demonstrate that our method effectively mitigates the challenges of noisy labels, often necessitating few to no validation samples, and is well generalized to other tasks such as image segmentation. This not only positions it as a robust complement to existing LNL techniques but also underscores its practical applicability. The code and models are available at https://github.com/yuyinzhou/l2b.

L2B: Learning to Bootstrap Robust Models for Combating Label Noise

TL;DR

A simple and effective method, named Learning to Bootstrap (L2B), which enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels, by dynamically adjusting the importance weight between real observed and generated labels.

Abstract

Deep neural networks have shown great success in representation learning. However, when learning with noisy labels (LNL), they can easily overfit and fail to generalize to new data. This paper introduces a simple and effective method, named Learning to Bootstrap (L2B), which enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels. It achieves this by dynamically adjusting the importance weight between real observed and generated labels, as well as between different samples through meta-learning. Unlike existing instance reweighting methods, the key to our method lies in a new, versatile objective that enables implicit relabeling concurrently, leading to significant improvements without incurring additional costs. L2B offers several benefits over the baseline methods. It yields more robust models that are less susceptible to the impact of noisy labels by guiding the bootstrapping procedure more effectively. It better exploits the valuable information contained in corrupted instances by adapting the weights of both instances and labels. Furthermore, L2B is compatible with existing LNL methods and delivers competitive results spanning natural and medical imaging tasks including classification and segmentation under both synthetic and real-world noise. Extensive experiments demonstrate that our method effectively mitigates the challenges of noisy labels, often necessitating few to no validation samples, and is well generalized to other tasks such as image segmentation. This not only positions it as a robust complement to existing LNL techniques but also underscores its practical applicability. The code and models are available at https://github.com/yuyinzhou/l2b.
Paper Structure (35 sections, 1 theorem, 23 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 35 sections, 1 theorem, 23 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Learning with noisy labels.
  4. Explicit relabeling.
  5. Instance reweighting.
  6. Meta-learning.
  7. Methodology
  8. Preliminary
  9. Learning to Bootstrap through Meta-Learning
  10. Online Approximation.
  11. Convergence Analysis
  12. Experiments
  13. Datasets
  14. CIFAR-10 & CIFAR-100.
  15. ISIC2019.
  16. ...and 20 more sections

Key Result

Theorem 1

Suppose that the training loss function $f,g$ have $\sigma$-bounded gradients and the validation loss $f^v$ is Lipschitz smooth with constant L. With a small enough learning rate $\lambda$, the validation loss monotonically decreases for any training batch $B$, namely, where $\theta_{t+1}$ is obtained using Eq. eq:theta_t+1 and $G$ is the validation loss Furthermore, Eq. eq:Validation_loss_decre

Figures (5)

  • Figure 1: (a) The original bootstrapping loss reed2014training is sensitive to the reweighting hyper-parameter $\beta$. Under different noise levels, the optimal $\beta$ is different (NF stands for noise fraction). (b) Schematic description of our Learning to Bootstrap (i.e., L2B) method. The reweighting hyper-parameters are learned in a meta-process.
  • Figure 2: Examples of $\alpha$ and $\beta$ on CIFAR-10 with asymmetric noise fraction of 20%. When the estimated pseudo label is of high-quality, i.e., the pseudo label is different from the noisy label but equal to the clean label, our model will automatically assign a much higher weight to $\beta$ than to $\alpha$ for corrupted training samples. When the pseudo label is equal to the noisy label (i.e., the two loss terms are equal to each other), $\alpha$ and $\beta$ are almost identical.
  • Figure 3: Comparison among different normalization functions (i.e., Eq. \ref{['eq:alpha_norm']}, Sigmoid function and Softmax function). Testing accuracy curve: (a) with different normalization functions under 40% symmetric noise label on the ISIC dataset. (b) with different normalization under 40% symmetric label noise on CIFAR-100.
  • Figure 4: Test accuracy v.s. number of epochs on CIFAR-100 under the noise fraction of 20% and 40%.
  • Figure 5: Visual comparison of prostate MRI images with noisy (contoured in yellow) and accurate (contoured in red) segmentation masks to demonstrate the discrepancy in segmentation quality between the two.

Theorems & Definitions (1)

  • Theorem