Space Alignment Matters: The Missing Piece for Inducing Neural Collapse in Long-Tailed Learning

TL;DR

This work identifies space misalignment between feature and classifier spaces as a missing piece for inducing Neural Collapse under long-tailed data. It introduces an Optimal Error Exponent framework to quantify how misalignment degrades class separability and then proposes three plug-and-play alignment strategies (SpA-Reg, SpA-SLERP, SpA-Proj) that integrate with existing long-tail methods without architectural changes. Empirical results on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT show consistent improvements and state-of-the-art performance, while recovery of NC properties in both feature and decision spaces is observed. The findings highlight space alignment as a crucial factor for robust minority-class generalization in imbalanced learning settings.

Abstract

Recent studies on Neural Collapse (NC) reveal that, under class-balanced conditions, the class feature means and classifier weights spontaneously align into a simplex equiangular tight frame (ETF). In long-tailed regimes, however, severe sample imbalance tends to prevent the emergence of the NC phenomenon, resulting in poor generalization performance. Current efforts predominantly seek to recover the ETF geometry by imposing constraints on features or classifier weights, yet overlook a critical problem: There is a pronounced misalignment between the feature and the classifier weight spaces. In this paper, we theoretically quantify the harm of such misalignment through an optimal error exponent analysis. Built on this insight, we propose three explicit alignment strategies that plug-and-play into existing long-tail methods without architectural change. Extensive experiments on the CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT datasets consistently boost examined baselines and achieve the state-of-the-art performances.

Figures (4)

• Figure 1: Space misalignment issue. (a) shows ResNet-32 model performances and corresponding cosine similarities between class feature means and classifier weights on the CIFAR-100 dataset. The baseline, aligned, and misaligned models are obtained by training with cross-entropy (CE), SpA-Reg (proposed method for alignment), and negative SpA-Reg loss, respectively. (b), (c) and (d) are toy examples with 2-dimensional features and 3 classes, which illustrate different geometric structures of the class feature means (black crosses) and the classifier weights (gray lines).
• Figure 2: Cosine similarity between the feature space and the decision space across $d$ classes under two imbalance factors (IF=$100$ and IF=$200$).
• Figure 3: A toy example illustrating the process of space alignment using our proposed SpA-Reg method. Each sphere shows the change of orientations of classifier weights (red arrows) and class feature means (black arrows) as training progressed. The top row visualizes the standard long-tail learning, i.e., training with cross-entropy loss, where significant misalignment between the classifier weights and the feature center persists throughout the whole training process. In contrast, for the bottom row, the classifier weights gradually aligned with the feature mean during training.
• Figure 4: Performances and corresponding cosine similarities between class feature means and classifier weights on the CIFAR-100 dataset with the imbalance factor of 200.

Theorems & Definitions (8)

• Theorem 1: Large-Deviations Error Exponent
• Theorem 2: Optimal Error Exponent (OEE)
• Theorem 3: Standard Simplex ETF Distance Properties
• Lemma 1
• Lemma 2
• Lemma 3
• Theorem 4: Error Exponent under simplex misalignment setting
• Theorem 5: Optimal Error Exponent Under Simplex Misalignment Setting