Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

Wanli Hong, Shuyang Ling

TL;DR

This work analyzes neural collapse (NC) under cross-entropy loss in an unconstrained feature model (UFM) when data are imbalanced. By convexifying the UFM via a nuclear-norm regularized prediction matrix, it establishes a complete picture of NC under imbalance, including a provable NC1 (within-class collapse) and the absence of NC2/NC3 in general, along with an ETF-like block structure within class clusters. A sharp minority-collapse threshold is derived for a two-cluster setting, and the analysis reveals that imbalance effects diminish asymptotically as total sample size grows, with mean predictions converging to a regular simplex ETF. Numerical experiments on standard vision datasets corroborate the theory and offer practical guidance on data oversampling to mitigate minority collapse. Collectively, the results extend NC theory to imbalanced data under CE and provide concrete insights into model regularization and data distribution design for robust generalization.

Abstract

Recent years have witnessed the huge success of deep neural networks (DNNs) in various tasks of computer vision and text processing. Interestingly, these DNNs with massive number of parameters share similar structural properties on their feature representation and last-layer classifier at terminal phase of training (TPT). Specifically, if the training data are balanced (each class shares the same number of samples), it is observed that the feature vectors of samples from the same class converge to their corresponding in-class mean features and their pairwise angles are the same. This fascinating phenomenon is known as Neural Collapse (N C), first termed by Papyan, Han, and Donoho in 2019. Many recent works manage to theoretically explain this phenomenon by adopting so-called unconstrained feature model (UFM). In this paper, we study the extension of N C phenomenon to the imbalanced data under cross-entropy loss function in the context of unconstrained feature model. Our contribution is multi-fold compared with the state-of-the-art results: (a) we show that the feature vectors exhibit collapse phenomenon, i.e., the features within the same class collapse to the same mean vector; (b) the mean feature vectors no longer form an equiangular tight frame. Instead, their pairwise angles depend on the sample size; (c) we also precisely characterize the sharp threshold on which the minority collapse (the feature vectors of the minority groups collapse to one single vector) will take place; (d) finally, we argue that the effect of the imbalance in datasize diminishes as the sample size grows. Our results provide a complete picture of the N C under the cross-entropy loss for the imbalanced data. Numerical experiments confirm our theoretical analysis.

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

TL;DR

This work analyzes neural collapse (NC) under cross-entropy loss in an unconstrained feature model (UFM) when data are imbalanced. By convexifying the UFM via a nuclear-norm regularized prediction matrix, it establishes a complete picture of NC under imbalance, including a provable NC1 (within-class collapse) and the absence of NC2/NC3 in general, along with an ETF-like block structure within class clusters. A sharp minority-collapse threshold is derived for a two-cluster setting, and the analysis reveals that imbalance effects diminish asymptotically as total sample size grows, with mean predictions converging to a regular simplex ETF. Numerical experiments on standard vision datasets corroborate the theory and offer practical guidance on data oversampling to mitigate minority collapse. Collectively, the results extend NC theory to imbalanced data under CE and provide concrete insights into model regularization and data distribution design for robust generalization.

Abstract

Recent years have witnessed the huge success of deep neural networks (DNNs) in various tasks of computer vision and text processing. Interestingly, these DNNs with massive number of parameters share similar structural properties on their feature representation and last-layer classifier at terminal phase of training (TPT). Specifically, if the training data are balanced (each class shares the same number of samples), it is observed that the feature vectors of samples from the same class converge to their corresponding in-class mean features and their pairwise angles are the same. This fascinating phenomenon is known as Neural Collapse (N C), first termed by Papyan, Han, and Donoho in 2019. Many recent works manage to theoretically explain this phenomenon by adopting so-called unconstrained feature model (UFM). In this paper, we study the extension of N C phenomenon to the imbalanced data under cross-entropy loss function in the context of unconstrained feature model. Our contribution is multi-fold compared with the state-of-the-art results: (a) we show that the feature vectors exhibit collapse phenomenon, i.e., the features within the same class collapse to the same mean vector; (b) the mean feature vectors no longer form an equiangular tight frame. Instead, their pairwise angles depend on the sample size; (c) we also precisely characterize the sharp threshold on which the minority collapse (the feature vectors of the minority groups collapse to one single vector) will take place; (d) finally, we argue that the effect of the imbalance in datasize diminishes as the sample size grows. Our results provide a complete picture of the N C under the cross-entropy loss for the imbalanced data. Numerical experiments confirm our theoretical analysis.
Paper Structure (26 sections, 12 theorems, 167 equations, 9 figures)

This paper contains 26 sections, 12 theorems, 167 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Notation
  3. Organization
  4. Preliminaries
  5. Unconstrained feature model
  6. Neural collapse
  7. ${\cal NC}$ on balanced datasets:
  8. ${\cal NC}$ on imbalanced data:
  9. ${\cal NC}$ beyond the UFMs:
  10. ${\cal NC}$ and training/generalization
  11. Main results
  12. Numerics
  13. Collapse of feature and prediction vectors
  14. Block structure of the mean prediction and features
  15. Minority collapse
  16. ...and 11 more sections

Key Result

Theorem 3.1

The global minimizer $(\boldsymbol{Z},\boldsymbol{b})$ to eq:cvx is unique and satisfies the following properties:

Figures (9)

  • Figure 1: Plot of $\log {\cal NC}_1$ v.s. epochs: $x$-axis is the epoch number and y-axis is $\log {\cal NC}_1$. Datasets: Cifar10 and FMNIST with two sets of parameters Dataset$_1$ and Dataset$_2$. Network: ResNet18 (red straight line), VGG11 (blue dotted line), and VGG13 (black dashed line).
  • Figure 2: Standardized (over $12$ matrices) mean prediction matrix $\bar{\boldsymbol{Z}}$ for all 12 experiments in Figure \ref{['fig:collapse']}. The white dashed lines separate clusters $A,B$ and $C$ in Dataset$_1$ and Dataset$_2$.
  • Figure 3: Relative error $\|\bar{\boldsymbol{Z}}-\bar{\boldsymbol{Z}}^*\|_{F}/\|\bar{\boldsymbol{Z}}^*\|_{F}$ ($\|\boldsymbol{b}-\boldsymbol{b}^*\|_2/\|\boldsymbol{b}^*\|_2$) and $\|\bar{\boldsymbol{Z}}-\bar{\boldsymbol{Z}}^*\|_{\infty}/\|\bar{\boldsymbol{Z}}^*\|_{\infty}$ ($\|\boldsymbol{b}-\boldsymbol{b}^*\|_{\infty}/\|\boldsymbol{b}^*\|_{\infty}$) v.s. the epoch for VGG13 on Cifar10 with Dataset$_1$ (Left) and VGG11 on Cifar10 with Dataset$_2$ (Right). The starting epoch numbers are chosen to be $400$ and $200$ when $\mathcal{NC}_1$ has reaches a low level.
  • Figure 4: Comparison of the min-max standardized final mean prediction matrix $\bar{\boldsymbol{Z}}$ and $\bar{\boldsymbol{Z}}^*$. Top: VGG13 on Cifar10 with Dataset$_1$; Bottom: VGG11 on Cifar10, with Dataset$_2$.
  • Figure 5: Plot for the mean prediction matrix $\bar{\boldsymbol{Z}}$ for $10$ classes with $k_A=k_B=5$ v.s. varying $\lambda_Z$. The white dashed lines separate clusters majority group $A$ and minority group $B$.
  • ...and 4 more figures

Theorems & Definitions (26)

  • Definition 3.1: Cluster structure
  • Theorem 3.1
  • Theorem 3.2: Block structure v.s. $\lambda_Z$
  • Corollary 3.3: Minority collapse threshold ($r\rightarrow \infty$, $n_B$ is fixed)
  • Theorem 3.4: $r = n_A/n_B > 1$ is fixed, $n_B\rightarrow \infty$
  • Theorem 3.5
  • Lemma 5.1
  • proof
  • Lemma 5.2: Optimality condition
  • proof
  • ...and 16 more