Class Uncertainty: A Measure to Mitigate Class Imbalance

TL;DR

This work argues that class imbalance cannot be fully captured by class cardinality alone and introduces Class Uncertainty, defined as the average predictive uncertainty of training examples estimated via Deep Ensembles. By aggregating per-example uncertainty into class-level measures, the authors demonstrate that Class Uncertainty correlates more strongly with class-wise difficulty and is robust to duplicates, enabling it to guide resampling, reweighting, margin adjustment, and multi-stage training across ten methods. Extensive experiments on long-tailed CIFAR-10/100 and a semantically imbalanced SVCI-20 dataset show that incorporating Class Uncertainty yields substantial or competitive gains versus cardinality-based baselines, with notable improvements on hard, under-represented or semantically difficult classes. The results suggest a practical path to more robust imbalance mitigation by focusing on predictive uncertainty, and the authors provide code and datasets to support further research, despite the higher computational cost of Deep Ensembles.

Abstract

Class-wise characteristics of training examples affect the performance of deep classifiers. A well-studied example is when the number of training examples of classes follows a long-tailed distribution, a situation that is likely to yield sub-optimal performance for under-represented classes. This class imbalance problem is conventionally addressed by approaches relying on the class-wise cardinality of training examples, such as data resampling. In this paper, we demonstrate that considering solely the cardinality of classes does not cover all issues causing class imbalance. To measure class imbalance, we propose "Class Uncertainty" as the average predictive uncertainty of the training examples, and we show that this novel measure captures the differences across classes better than cardinality. We also curate SVCI-20 as a novel dataset in which the classes have equal number of training examples but they differ in terms of their hardness; thereby causing a type of class imbalance which cannot be addressed by the approaches relying on cardinality. We incorporate our "Class Uncertainty" measure into a diverse set of ten class imbalance mitigation methods to demonstrate its effectiveness on long-tailed datasets as well as on our SVCI-20. Code and datasets will be made available.

Figures (4)

• Figure 1: (a) Using our Class Uncertainty improves top-1 error rate of all aforementioned methods (CIFAR-100-LT with an IR of 50 using ResNet-32). In particular, we obtain (i) the sampling probability of each class in resampling methods; (ii) the weights of the loss for each class; (iii) the margins to be enforced around each class; or (iv) again sampling probability in the second-stage of multi-stage training strategy using Class Uncertainty. The numbers on top of the histograms show relative gain over cardinality-based methods. Baselines: Progressively-balanced sampling kang2019decoupling, class-balanced Focal Loss cui2019classbalancedloss, LDAM with reweighting ldam and deferred resampling. (b) Naïve classifier is not robust to class imbalance, giving rise to a plethora of mitigation methods: (1) "resampling" samples a balanced set of examples; (2) "reweighting" assigns a weight for each class ($w_c^{(i)}$); (3) "margin-adjustment" methods assign different margins ($\Delta_c^{(i)}$) to the logits ($s_c^{(i)}$) of different classes; and (4) "multi-stage training" first trains a naïve classifiers followed by a resampling or reweighting method to re-train or fine-tune the classifier. These methods generally rely on the cardinality of the classes.
• Figure 2: (a,b)Class Cardinality Imbalance Measure ($\mu^{C}_c$) vs. Class Uncertainty ($\mu^{{U}}_c$) on CIFAR-10-LT and CIFAR-100-LT with an IR of 50. While $\mu^{C}_c$ is entirely determined by cardinality (from left to right, class cardinality decreases), $\mu^{{U}}_c$ presents more diverse values. (c-h) An analysis of $\mu^{C}_c$ and $\mu^{{U}}_c$ in terms of important features. For capturing the amount of imbalance among classes ((c) and (f)), Spearman correlation coefficient ($\rho$) between the imbalance measure and the test accuracy of a naïve classifier on CIFAR-100-LT is notably higher for $\mu^{{U}}_c$ compared to $\mu^{C}_c$, which implies that $\mu^{{U}}_c$ captures imbalance better. As IR increases, both can capture the change in IR in (d) and (g) by promoting the tail classes (in red) or demoting the head ones (in blue), while the relative change across classes in $\mu^{C}_c$ is more drastic. In (e) and (h), $\mu^{C}_c$ is highly sensitive to the duplicated training examples, which do not contain new information. $\mu^{{U}}_c$ is relatively less sensitive especially after the duplication strength of $0.3$. Therefore, $\mu^{{U}}_c$ is a better alternative to $\mu^{C}_c$ when considering these important features.
• Figure 3: (a) Conventional imbalance mitigation methods ignore variance of samples while estimating the decision boundary. (b) Our motivation is to promote the learning of classes with higher uncertainty, effectively pushing the decision boundary away from such classes. (c) High-level overview of our strategy. We first obtain Class Uncertainty ($\mu^{{U}}_c$) as the measure of imbalance using the predictive uncertainties of training examples based on DE. Then, we incorporate the resulting class imbalance measure $\mu^{{U}}_c$ into various methods during training the classifier.
• Figure 4: (a) As the number of examples from each class ($N_c$) decreases, the error gap between CIFAR-10 and SVHN classes in SVCI-20 dataset increases for Naïve Classifier; implying each setting corresponds to a different semantic imbalance level. (b) While Class Cardinality Imbalance Measure ($\mu^{C}_c$) does not prioritize any class on SVCI-20, our Class Uncertainty ($\mu^{{U}}_c$) differentiates between them. Generally, $\mu^{{U}}_c$ assigns more priority to CIFAR-10 classes (class index less than or equal to 9) as the harder classes.

Theorems & Definitions (3)

• Definition 3.1: Class Imbalance Measure
• Definition 3.2: Class Cardinality Imbalance Measure
• Definition 3.3: Class Uncertainty Imbalance Measure