Pushing Boundaries: Mixup's Influence on Neural Collapse

TL;DR

This paper investigates how mixup data augmentation shapes the geometry of last-layer activations in deep networks, asking whether mixup induces a distinct configuration beyond Neural Collapse. Through extensive empirical analysis across architectures and datasets, the authors discover that same-class mixup activations align with a simplex ETF along the classifier directions, while different-class activations form channels on the decision boundary, with earlier layers showing manifold-like behavior. They support these observations with a theoretical analysis using an unconstrained-features model under a simplex ETF classifier, showing that same-class features are lambda-independent and align with the classifier, whereas different-class features depend on the mixup coefficient and lie in linear combinations of mixed targets. Additional experiments with a fixed ETF classifier indicate the empirical features can align more closely with the theoretical optimum without sacrificing accuracy, and calibration gains from mixup are explained through this geometric structure. Collectively, the work provides a principled view of how mixup calibrates models by organizing last-layer activations and offers guidance for designing augmentation schemes and interpreting representation geometry.

Abstract

Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to augment the robustness and calibration of deep neural networks. Despite its widespread adoption, the nuanced mechanisms that underpin its success are not entirely understood. The observed phenomenon of Neural Collapse, where the last-layer activations and classifier of deep networks converge to a simplex equiangular tight frame (ETF), provides a compelling motivation to explore whether mixup induces alternative geometric configurations and whether those could explain its success. In this study, we delve into the last-layer activations of training data for deep networks subjected to mixup, aiming to uncover insights into its operational efficacy. Our investigation, spanning various architectures and dataset pairs, reveals that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect. In this configuration, activations from mixed-up examples of identical classes align with the classifier, while those from different classes delineate channels along the decision boundary. Moreover, activations in earlier layers exhibit patterns, as if trained with manifold mixup. These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss). By analyzing this distinctive geometric configuration, we elucidate the mechanisms by which mixup enhances model calibration. To further validate our empirical observations, we conduct a theoretical analysis under the assumption of an unconstrained features model, utilizing the mixup loss. Through this, we characterize and derive the optimal last-layer features under the assumption that the classifier forms a simplex ETF.

Figures (12)

• Figure 1: (Visualization of activations outputted by networks trained with mixup). Last-layer activations of mixup training data for a randomly selected subset of three classes across various dataset and network architecture combinations trained with mixup. The first row illustrates activations generated by a WideResNet, while the second row showcases activations from a ViT. Each column corresponds to a different dataset. Coloration indicates the type of mixup (same or different class), along with the level of mixup, $\lambda$. For each plot, the relevant classifiers are plotted in black.
• Figure 2: (Visualization of activations outputted by networks trained with various loss functions). Last-layer activations for WideResNet-40-10 trained on the CIFAR10 dataset, subsetted to three randomly selected classes. Projections are generated using the same method as Figure \ref{['fig:features']}. Left to right: baseline cross-entropy, MSE mixup, cross-entropy mixup. Colouring indicates mixup type (same-class or different-class), and the level of mixup, $\lambda$. Relevant classifiers plotted in black. Additional dataset architecture combinations for baseline cross-entropy are available in appendix \ref{['sec:additiona_plots']}.
• Figure 3: (Projection of CLS token at each layer). Projections of the CLS token of the mix up of two randomly selected training images for various values of $\lambda$. Trajectories start at the origin. Colouring indicates mixup type (same-class or different-class), and the level of mixup, $\lambda$.
• Figure 4: (Diagram showing the relationship between calibration and the configuration). As $\lambda$ approaches 0.5, the last-layer activation $h_{i,i^\prime}^\lambda$ (black) traverses the blue line of the configuration, leading to less confident predictions. Simultaneously, the variability of the activation (perforated black circle) results in an increase in misclassification due to the probability of being on the incorrect side of the decision boundary (green) increasing.
• Figure 5: (Convergence of classifier to simplex ETF). Measurements on the classifier, ${\bm{W}}$, for each network architecture and dataset combination. First and third plot: Coefficient of variation of the classifier norms, $\operatorname{Std}_i\left(\left\|{\bm{w}}_i\right\|_2\right) / \operatorname{Avg}_i\left(\left\|{\bm{w}}_i\right\|_2\right)$. Second and fourth plot: Standard deviation of the cosines between classifiers of distinct classes, $\operatorname{Std}_{i, i^{\prime} \neq i} \left(\langle{\bm{w}}_i, {\bm{w}}_{i^{\prime}}\rangle /\left(\left\|{\bm{w}}_i\right\|_2\left\|{\bm{w}}_{i^{\prime}}\right\|_2\right)\right)$ with $i \neq i^\prime$. As training progresses, measurements indicate that ${\bm{W}}$ is trending toward a simplex ETF configuration.

Theorems & Definitions (2)

• Theorem 3.1
• proof