Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Impact of Label Noise on Learning Complex Features

Rahul Vashisht, P. Krishna Kumar, Harsha Vardhan Govind, Harish G. Ramaswamy

TL;DR

It is shown that pretraining promotes learning complex functions and diverse features in the presence of noise, and encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.

Abstract

Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.

Impact of Label Noise on Learning Complex Features

TL;DR

It is shown that pretraining promotes learning complex functions and diverse features in the presence of noise, and encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.

Abstract

Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.

Paper Structure

This paper contains 12 sections, 12 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Effect of Noisy Pre-training on Learned Decision Functions
  3. Experiments
  4. Discussion and Conclusion
  5. Appendix
  6. Dataset Description:
  7. Network Architectures
  8. Additional Results:
  9. 4D Slab Data Decision Boundary plots for Different Random Seeds
  10. Impact of Label Smoothing
  11. Varying Model Architecture
  12. Visualizing Feature Importance Using Top Eigenvectors

Figures (12)

  • Figure 1: (Top-left) First 4 dimensions of Slab-data. (Top-right) Decision boundary learned from regular training. (Botton row) Two different decision boundaries learned after noisy pre-training. All models achieve $100$% training and test accuracy.
  • Figure 2: MNIST-FMNIST dominoes data samples with visualizing first layer neural feature matrix $W_1^{\top}W_1$, for (Left) Standard training, and (Right) Random pre-training respectively.
  • Figure 3: Samples from Waterbirds dataset. We visualize the diagonal of first layer neural feature matrix $W_1^{\top}W_1$ for (Left) standard training and (Right) random pretraining.
  • Figure 4: Noise augmented multi-slab dataset
  • Figure 5: Noise augmented multi-slab dataset
  • ...and 7 more figures