Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup

Muthu Chidambaram, Xiang Wang, Chenwei Wu, Rong Ge

TL;DR

The main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class.

Abstract

Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or views) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have multiple features.

Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup

TL;DR

The main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class.

Abstract

Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or views) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have multiple features.
Paper Structure (21 sections, 27 theorems, 114 equations, 2 figures, 1 table)

This paper contains 21 sections, 27 theorems, 114 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Work
  4. Background on Mixup
  5. Motivating Midpoint Mixup: The Linear Regime
  6. Midpoint Mixup with Linear Models on Linearly Separable Data
  7. Analyzing Midpoint Mixup Training Dynamics on General Multi-View Data
  8. General Multi-View Data Setup
  9. Main Results
  10. Experiments
  11. Conclusion
  12. Supporting Lemmas and Calculations
  13. Gaussian Concentration and Anti-concentration Results
  14. Gradient Calculations
  15. Proofs of Main Results
  16. ...and 6 more sections

Key Result

Lemma 3.1

[Midpoint Mixup Optimal Direction] A linear model $g$ satisfies the following if $g$ has the property that for every class $y$ we have $\left\langle w_y, v_{y, \ell_1} \right\rangle = \left\langle w_s, v_{s, \ell_2} \right\rangle > 0$ and $\left\langle w_y, v_{s, \ell_2} \right\rangle = 0$ for every $s \neq y$ and $\ell_1, \ell_2 \in [2]$. Furthermore, with probability $1 - \

Figures (2)

  • Figure 1: Visualization of data modification in CIFAR-10.
  • Figure 2: Test error comparison between Uniform Mixup (green), Midpoint Mixup (orange), and ERM (blue). Each curve represents the average of 5 model runs (over the randomness of the data augmentations and model initializations), while the surrounding area represents 1 standard deviation.

Theorems & Definitions (92)

  • Definition 3.1
  • Lemma 3.1
  • Proposition 3.1
  • Proposition 3.1
  • Proposition 3.1
  • Definition 4.1
  • Definition 4.2
  • Proposition 4.2
  • Definition 4.3
  • Theorem 4.3
  • ...and 82 more