Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding Why Data Augmentation Improves Generalization

Jingyang Li, Jiachun Pan, Kim-Chuan Toh, Pan Zhou

TL;DR

This work addresses the lack of a unified theory for data augmentation by identifying two fundamental mechanisms—partial semantic feature removal and feature mixing—that underlie many augmentations. The authors develop a theoretical framework demonstrated on a three-layer CNN, proving that A1 promotes diverse feature learning, A2 promotes robust feature learning, and A3 combines both for superior generalization. They validate the theory with experiments on CIFAR-100 and Tiny-ImageNet, showing that augmentations implementing A3 (e.g., CutMix, SaliencyMix) achieve the best generalization, especially under noisy conditions. The framework offers a principled basis for designing augmentation strategies and has implications for improving generalization across vision tasks.

Abstract

Data augmentation is a cornerstone technique in deep learning, widely used to improve model generalization. Traditional methods like random cropping and color jittering, as well as advanced techniques such as CutOut, Mixup, and CutMix, have achieved notable success across various domains. However, the mechanisms by which data augmentation improves generalization remain poorly understood, and existing theoretical analyses typically focus on individual techniques without a unified explanation. In this work, we present a unified theoretical framework that elucidates how data augmentation enhances generalization through two key effects: partial semantic feature removal and feature mixing. Partial semantic feature removal reduces the model's reliance on individual feature, promoting diverse feature learning and better generalization. Feature mixing, by scaling down original semantic features and introducing noise, increases training complexity, driving the model to develop more robust features. Advanced methods like CutMix integrate both effects, achieving complementary benefits. Our theoretical insights are further supported by experimental results, validating the effectiveness of this unified perspective.

Towards Understanding Why Data Augmentation Improves Generalization

TL;DR

This work addresses the lack of a unified theory for data augmentation by identifying two fundamental mechanisms—partial semantic feature removal and feature mixing—that underlie many augmentations. The authors develop a theoretical framework demonstrated on a three-layer CNN, proving that A1 promotes diverse feature learning, A2 promotes robust feature learning, and A3 combines both for superior generalization. They validate the theory with experiments on CIFAR-100 and Tiny-ImageNet, showing that augmentations implementing A3 (e.g., CutMix, SaliencyMix) achieve the best generalization, especially under noisy conditions. The framework offers a principled basis for designing augmentation strategies and has implications for improving generalization across vision tasks.

Abstract

Data augmentation is a cornerstone technique in deep learning, widely used to improve model generalization. Traditional methods like random cropping and color jittering, as well as advanced techniques such as CutOut, Mixup, and CutMix, have achieved notable success across various domains. However, the mechanisms by which data augmentation improves generalization remain poorly understood, and existing theoretical analyses typically focus on individual techniques without a unified explanation. In this work, we present a unified theoretical framework that elucidates how data augmentation enhances generalization through two key effects: partial semantic feature removal and feature mixing. Partial semantic feature removal reduces the model's reliance on individual feature, promoting diverse feature learning and better generalization. Feature mixing, by scaling down original semantic features and introducing noise, increases training complexity, driving the model to develop more robust features. Advanced methods like CutMix integrate both effects, achieving complementary benefits. Our theoretical insights are further supported by experimental results, validating the effectiveness of this unified perspective.

Paper Structure

This paper contains 48 sections, 10 theorems, 173 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Augmentation
  4. Feature Learning Analysis
  5. Problem Setup
  6. Data Distribution
  7. Supervised Learning with Data Augmentation
  8. Neural Network
  9. Supervised Learning
  10. Data Augmentation
  11. Main Results
  12. Diverse Feature Learning with $\mathcal{A}_1$
  13. Robust Feature Learning with $\mathcal{A}_2$
  14. Diverse & Robust Feature Learning via $\mathcal{A}_3$
  15. Discussion
  16. ...and 33 more sections

Key Result

Theorem 4.3

Suppose Assumptions assum:a1 and assum1 holds. For sufficiently large $k$ and $m = \mathrm{polylog}(k)$, by setting $\eta \leq 1 / \mathrm{poly}(k)$, after independently running SL with $\mathcal{A}_1$ and vanilla SL without $\mathcal{A}_1$ for both $T = \mathrm{poly}(k) / \eta$ iterations, the foll But for vanilla SL without $\mathcal{A}_1$ , there is an $l \in [2]$ such that: (b) Improved Gener

Figures (2)

  • Figure 1: Visualization of pretrained ResNet-50 he2016deep using Grad-CAM on airplane images from ImageNet.
  • Figure 2: Visualization of data augmentation effects on ImageNet images. (a) Partial semantic feature removal ($\mathcal{A}_1$): original image (left) and augmented version (right) are shown for each pair. (b) Feature mixing ($\mathcal{A}_2$): original images (left and middle) blended into the augmented version (right). (c) Combined effect ($\mathcal{A}_3$): original images (left and middle) together create the augmented version (right).

Theorems & Definitions (41)

  • Definition 3.1: Informal, Data Distribution allen-zhu2023towards
  • Theorem 4.3
  • Lemma 4.5
  • Theorem 4.6
  • Theorem 4.8
  • Definition 1.1: Data distributions for single-view $\mathcal{D}_s$ and multi-view data $\mathcal{D}_m$ allen-zhu2023towards
  • Definition 1.2: Training dataset
  • Definition 1.3: Noisy data distribution $\mathcal{D}_{\text{noisy}}$
  • Proposition 2.1
  • Definition 4.2
  • ...and 31 more