For Better or For Worse? Learning Minimum Variance Features With Label Augmentation

TL;DR

This work investigates why label augmentation methods like label smoothing and Mixup improve performance, by analyzing how they shape feature learning. It provides a precise theory: in linear binary classification with a split into low-variance ($\mathcal{L}$) and high-variance ($\mathcal{H}$) features, label augmentation drives models to rely on $\mathcal{L}$, while standard weight decay pushes toward $\mathcal{H}$. The authors extend to multi-class settings, deriving variance-based lower bounds for LS and Mixup losses and showing that reducing model-output variance is essential to improve these losses. Empirically, LS and Mixup reduce activation and output variances and often improve generalization on CIFAR benchmarks, but they can also magnify vulnerability to spurious low-variance correlations, as shown in synthetic and colored-MMNIST-like experiments. The findings highlight a principled trade-off: variance-minimizing learning can yield strong performance in typical tasks but may compromise robustness to spuriously correlated cues, motivating future work on variance-aware regularization and robust feature learning.

Abstract

Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data augmentation techniques - which includes both label smoothing and Mixup - involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models on binary classification data trained with label augmentation learn only the minimum variance features in the data, while standard training (which includes weight decay) can learn higher variance features. We then use our techniques to show that even for nonlinear models and general data distributions, the label smoothing and Mixup losses are lower bounded by a function of the model output variance. Lastly, we demonstrate empirically that this aspect of label smoothing and Mixup can be a positive and a negative. On the one hand, we show that the strong performance of label smoothing and Mixup on image classification benchmarks is correlated with learning low variance hidden representations. On the other hand, we show that Mixup and label smoothing can be more susceptible to low variance spurious correlations in the training data.

Figures (8)

• Figure 1: ResNet-18 final test errors, penultimate layer activation variances, and output probability variances on CIFAR-10 and CIFAR-100. Activation variance results are shown starting at epoch 25 as early epochs have larger scale oscillations in the computed variance.
• Figure 2: Logistic regression final test errors for various hyperparameter settings on the binary classification versions of CIFAR-10 and CIFAR-100 from Section \ref{['sec:binary']}.
• Figure 3: Comparison of norm ratio between first dimension (synthetically modified in the training data) and remaining dimensions (left unchanged) for trained logistic regression weight vector for the 20 different hyperparameter settings of weight decay, label smoothing, and Mixup.
• Figure 4: Final model test errors over our hyperparameter sweep for the colored MNIST dataset of Section \ref{['sec:background']}, alongside a visualization of samples from the dataset.
• Figure 5: Visualization of weight decay, label smoothing, and Mixup decision boundaries. Figure (a) considers canonical hyperparameter choices, and Figures (b) and (c) illustrate the effects of scaling these choices.

Theorems & Definitions (26)

• Definition 2.0
• Definition 3.0
• Theorem 3.1
• Corollary 3.1
• Theorem 3.2
• Theorem 3.3
• Definition 3.3
• Proposition 3.3
• Proposition 3.3
• Remark 3.4