Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Exploration of Neural Collapse under Imbalanced Data

Haixia Liu

TL;DR

This paper considers the $L-extended unconstrained feature model with a bias term and provides a theoretical analysis of global minimizer, finding that features within the same class converge to their class mean, similar to both the balanced case and the imbalanced case without bias.

Abstract

Neural collapse, a newly identified characteristic, describes a property of solutions during model training. In this paper, we explore neural collapse in the context of imbalanced data. We consider the $L$-extended unconstrained feature model with a bias term and provide a theoretical analysis of global minimizer. Our findings include: (1) Features within the same class converge to their class mean, similar to both the balanced case and the imbalanced case without bias. (2) The geometric structure is mainly on the left orthonormal transformation of the product of $L$ linear classifiers and the right transformation of the class-mean matrix. (3) Some rows of the left orthonormal transformation of the product of $L$ linear classifiers collapse to zeros and others are orthogonal, which relies on the singular values of $\hat Y=(I_K-1/N\mathbf{n}1^\top_K)D$, where $K$ is class size, $\mathbf{n}$ is the vector of sample size for each class, $D$ is the diagonal matrix whose diagonal entries are given by $\sqrt{\mathbf{n}}$. Similar results are for the columns of the right orthonormal transformation of the product of class-mean matrix and $D$. (4) The $i$-th row of the left orthonormal transformation of the product of $L$ linear classifiers aligns with the $i$-th column of the right orthonormal transformation of the product of class-mean matrix and $D$. (5) We provide the estimation of singular values about $\hat Y$. Our numerical experiments support these theoretical findings.

The Exploration of Neural Collapse under Imbalanced Data

TL;DR

This paper considers the $L-extended unconstrained feature model with a bias term and provides a theoretical analysis of global minimizer, finding that features within the same class converge to their class mean, similar to both the balanced case and the imbalanced case without bias.

Abstract

Neural collapse, a newly identified characteristic, describes a property of solutions during model training. In this paper, we explore neural collapse in the context of imbalanced data. We consider the -extended unconstrained feature model with a bias term and provide a theoretical analysis of global minimizer. Our findings include: (1) Features within the same class converge to their class mean, similar to both the balanced case and the imbalanced case without bias. (2) The geometric structure is mainly on the left orthonormal transformation of the product of linear classifiers and the right transformation of the class-mean matrix. (3) Some rows of the left orthonormal transformation of the product of linear classifiers collapse to zeros and others are orthogonal, which relies on the singular values of , where is class size, is the vector of sample size for each class, is the diagonal matrix whose diagonal entries are given by . Similar results are for the columns of the right orthonormal transformation of the product of class-mean matrix and . (4) The -th row of the left orthonormal transformation of the product of linear classifiers aligns with the -th column of the right orthonormal transformation of the product of class-mean matrix and . (5) We provide the estimation of singular values about . Our numerical experiments support these theoretical findings.

Paper Structure

This paper contains 16 sections, 8 theorems, 66 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Setup
  4. Theoretical Analysis
  5. Unconstrained Feature Model
  6. -Extended Unconstrained Feature Model
  7. The Estimation of Singular Values
  8. Numerical Results
  9. Conclusion
  10. Auxiliary Lemma
  11. Technical Details in Proof
  12. Proof of Theorem \ref{['theo:ufm-biasfree']}
  13. Proof of Theorem \ref{['theo:ufm-bias']}
  14. Proof of Theorem \ref{['theo:bias_mult-layer']}
  15. Proof of Theorem \ref{['theo:singular']}
  16. ...and 1 more sections

Key Result

Theorem 4.1

We consider a dataset with $K$ classes, and there are $n_k$ samples in the $k$-th class for $k=1,\cdots,K$. Define $D=\mathrm{diag}([\sqrt{n_1},\cdots,\sqrt{n_K}])$. Let $W=^\top\in{\mathbb R}^{K\times d}$ with $W_k\in{\mathbb R}^d$, $H=\in{\mathbb R}^{d\times N}$ with $H_k\in{\mathbb R}^{d\times n_

Figures (4)

  • Figure 1: Bias Case: The performance of the NC metrics and training accuracy versus epoch with a 6-layer MLP backbone on an imbalanced subset of CIFAR10 for $L$-extended unconstrained feature model with $L=1,3,6$.
  • Figure 2: Base case: The performance of the NC metrics and training accuracy versus epoch with a 6-layer MLP backbone on an imbalanced subset of EMNIST for $L$-extended unconstrained feature model with $L=1,3,6$.
  • Figure 3: Bias-free case: The performance of the NC metrics and training accuracy versus epoch with a 6-layer MLP backbone on an imbalanced subset of CIFAR10 for $L$-extended unconstrained feature model with $L=1,3,6$.
  • Figure 4: Bias-free case: The performance of the NC metrics and training accuracy versus epoch with a 6-layer MLP backbone on an imbalanced subset of EMNIST for $L$-extended unconstrained feature model with $L=1,3,6$.

Theorems & Definitions (10)

  • Definition 3.1: $L$-extended unconstrained feature model
  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Definition 4.1: $(\{N_i,\ell_i\}^m_{i=1},\{n_j\}^K_{j=1})$-Imbalances
  • Theorem 4.4
  • Lemma A.1: Lemma 2.3 in zhu2021geometric
  • Lemma A.2: Minimizer of the function $g(x)=\frac{1}{ x^L+1}+\alpha x$, D.2.1 in dang2023neural
  • Theorem B.1
  • Theorem B.2