BCE vs. CE in Deep Feature Learning

TL;DR

This work challenges the conventional view that neural collapse (NC) is exclusive to cross-entropy (CE) loss by proving that binary cross-entropy (BCE) can also drive NC under mild conditions, with BCE’s classifier biases playing a central, explicit role in shaping feature distributions. The authors contrast CE and BCE at both theoretical and practical levels, showing BCE yields explicit, uniform constraints on decision scores that promote intra-class compactness and inter-class distinctiveness, often translating to improved test accuracy and uniform accuracy across datasets. Empirically, BCE accelerates NC during training and delivers stronger feature properties (as measured by compactness and distinctiveness) and better practical performance on MNIST, CIFAR-10/100, ImageNet, and long-tailed CIFAR-100, while remaining robust to variations in batch size and augmentation. The results position BCE as a strong alternative to CE for deep feature learning in multi-class tasks, with implications for both CNNs and Transformers.

Abstract

When training classification models, it expects that the learned features are compact within classes, and can well separate different classes. As the dominant loss function for training classification models, minimizing cross-entropy (CE) loss maximizes the compactness and distinctiveness, i.e., reaching neural collapse (NC). The recent works show that binary CE (BCE) performs also well in multi-class tasks. In this paper, we compare BCE and CE in deep feature learning. For the first time, we prove that BCE can also maximize the intra-class compactness and inter-class distinctiveness when reaching its minimum, i.e., leading to NC. We point out that CE measures the relative values of decision scores in the model training, implicitly enhancing the feature properties by classifying samples one-by-one. In contrast, BCE measures the absolute values of decision scores and adjust the positive/negative decision scores across all samples to uniformly high/low levels. Meanwhile, the classifier biases in BCE present a substantial constraint on the decision scores to explicitly enhance the feature properties in the training. The experimental results are aligned with above analysis, and show that BCE could improve the classification and leads to better compactness and distinctiveness among sample features. The codes will be released.

Figures (7)

• Figure 1: The feature distributions of CE and BCE losses in the distance space. We respectively apply the blue, red, and green shading to indicate the feature regions of three categories. The pentagrams represent their classifiers, and the solid dot represents a general feature $\bm h^{(2)}$ in the second category. Since the distance between two vectors is inversely proportional to their similarity/inner product, CE loss requires the distance from the feature to its classifier vector to be less than the distance to other classifier vectors, while BCE loss requires the distance to be less than its corresponding bias. Small $b'_k$ implies large $b_k$ in Eq. (\ref{['eq_ideal_BCE_pos_neg']}).
• Figure 2: NC metrics of ResNet18 trained on CIFAR10 with CE and BCE using SGD and AdamW, respectively. The NC metrics approach zero at the terminal phase of training, while the NC metrics of BCE decrease faster than that of CE in the first 20 epochs.
• Figure 3: The distributions of the final classifier bias and positive/negative decision scores for 60 ResNet18s trained on MNIST with fixed weight decay factor $\lambda_{\bm b}$ (top) and varying $\lambda_{\bm b}$ (bottom), while $\lambda_{\bm W} = \lambda_{\bm H} = 5\times10^{-4}$. The mean $\bar{b} = \frac{1}{K}\sum_{k=1}^K b_k$ of initialized biases is respectively set as $0,1,2,3,4,5,6,8,10$ in the experiments with fixed $\lambda_{\bm b} = 0$, and the bias mean is set as $10$ in that with varying $\lambda_{\bm b}$.
• Figure 4: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle), DenseNet121 (bottom) on MNIST with CE and BCE using SGD and AdamW, respectively, with $\lambda_{\bm W} = \lambda_{\bm H} = \lambda_{\bm b} = 5\times10^{-4}$.
• Figure 5: The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle), DenseNet121 (bottom) on CIFAR10 with CE and BCE using SGD and AdamW, respectively, with $\lambda_{\bm W} = \lambda_{\bm H} = \lambda_{\bm b} = 5\times10^{-4}$.

Theorems & Definitions (11)

• Theorem 3.1
• Theorem 3.2
• Definition 3.1
• Theorem 3.2
• Theorem 3.3
• Lemma 3.4
• Lemma 3.5
• Lemma 3.6
• Lemma 3.7
• Lemma 3.8