Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reimplementation of Learning to Reweight Examples for Robust Deep Learning

Parth Patil, Ben Boardley, Jack Gardner, Emily Loiselle, Deerajkumar Parthipan

TL;DR

This work reimplements the Learning to Reweight Examples framework to enhance robustness of deep models under label noise and dataset bias. By leveraging a small, clean validation set, it online-adjusts per-example weights via gradient-based meta-learning, converting a nested optimization into an efficient online procedure with convergence guarantees. Empirical evaluation on a toy imbalanced CIFAR-10 task and the HAM10000 skin-lesion dataset shows improved robustness and class-distribution performance over standard SGD and simple baselines, though broader competitiveness with top challenge results requires further resources. The approach offers a practical, validation-guided alternative to offline hyperparameter tuning for noisy and imbalanced data scenarios, with theoretical convergence assurances and explicit second-order-gradient computations.

Abstract

Deep neural networks (DNNs) have been used to create models for many complex analysis problems like image recognition and medical diagnosis. DNNs are a popular tool within machine learning due to their ability to model complex patterns and distributions. However, the performance of these networks is highly dependent on the quality of the data used to train the models. Two characteristics of these sets, noisy labels and training set biases, are known to frequently cause poor generalization performance as a result of overfitting to the training set. This paper aims to solve this problem using the approach proposed by Ren et al. (2018) using meta-training and online weight approximation. We will first implement a toy-problem to crudely verify the claims made by the authors of Ren et al. (2018) and then venture into using the approach to solve a real world problem of Skin-cancer detection using an imbalanced image dataset.

Reimplementation of Learning to Reweight Examples for Robust Deep Learning

TL;DR

This work reimplements the Learning to Reweight Examples framework to enhance robustness of deep models under label noise and dataset bias. By leveraging a small, clean validation set, it online-adjusts per-example weights via gradient-based meta-learning, converting a nested optimization into an efficient online procedure with convergence guarantees. Empirical evaluation on a toy imbalanced CIFAR-10 task and the HAM10000 skin-lesion dataset shows improved robustness and class-distribution performance over standard SGD and simple baselines, though broader competitiveness with top challenge results requires further resources. The approach offers a practical, validation-guided alternative to offline hyperparameter tuning for noisy and imbalanced data scenarios, with theoretical convergence assurances and explicit second-order-gradient computations.

Abstract

Deep neural networks (DNNs) have been used to create models for many complex analysis problems like image recognition and medical diagnosis. DNNs are a popular tool within machine learning due to their ability to model complex patterns and distributions. However, the performance of these networks is highly dependent on the quality of the data used to train the models. Two characteristics of these sets, noisy labels and training set biases, are known to frequently cause poor generalization performance as a result of overfitting to the training set. This paper aims to solve this problem using the approach proposed by Ren et al. (2018) using meta-training and online weight approximation. We will first implement a toy-problem to crudely verify the claims made by the authors of Ren et al. (2018) and then venture into using the approach to solve a real world problem of Skin-cancer detection using an imbalanced image dataset.
Paper Structure (21 sections, 22 equations, 4 figures, 2 tables)

This paper contains 21 sections, 22 equations, 4 figures, 2 tables.

Table of Contents

  1. Problem Statement
  2. Method
  3. Related Work
  4. Learning to reweight examples in a MLP (Multi-Layer Perceptron Network)
  5. Implementation using Automatic Differentiation
  6. Proof of Convergence of the Reweighted Training
  7. Experiment
  8. Toy Problem: Implementation Summary
  9. Toy Problem: Dataset
  10. Toy Problem: Implementation Details
  11. Toy Problem: Results
  12. Toy Problem: Conclusion
  13. Real-world Problem Description
  14. Real-world Dataset
  15. Real-world: Intended Outcome
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Accuracy of Model Trained with SGD vs Model Trained with Reweighting Algorithm
  • Figure 2: Distribution of training dataset
  • Figure 3: Confusion Matrix for Unweighted Training
  • Figure 4: Confusion Matrix for Learning to Reweight Training