The Hidden Influence of Latent Feature Magnitude When Learning with Imbalanced Data

TL;DR

The paper investigates how latent feature magnitudes in the final inference layer influence generalization under class imbalance. By analyzing Classification Embeddings ($CE$) across CNN, SVM, and LG classifiers, it shows that predictions are driven by the sum of a small set of high-magnitude features, limiting minority-class generalization even when data augmentation is used. The study demonstrates that only a minority of $CE$ are needed to predict individual instances, that top-$k$ features disproportionately drive decisions, and that frequency of feature usage correlates with magnitude in image tasks. These findings reveal a hidden bottleneck in imbalanced learning and suggest that improving generalization requires addressing the embedding and inference dynamics, not just data augmentation. The results have practical implications for designing more robust representations and inference mechanisms in imbalanced settings.

Abstract

Machine learning (ML) models have difficulty generalizing when the number of training class instances are numerically imbalanced. The problem of generalization in the face of data imbalance has largely been attributed to the lack of training data for under-represented classes and to feature overlap. The typical remedy is to implement data augmentation for classes with fewer instances or to assign a higher cost to minority class prediction errors or to undersample the prevalent class. However, we show that one of the central causes of impaired generalization when learning with imbalanced data is the inherent manner in which ML models perform inference. These models have difficulty generalizing due to their heavy reliance on the magnitude of encoded signals. During inference, the models predict classes based on a combination of encoded signal magnitudes that linearly sum to the largest scalar. We demonstrate that even with aggressive data augmentation, which generally improves minority class prediction accuracy, parametric ML models still associate a class label with a limited number of feature combinations that sum to a prediction, which can affect generalization.

Figures (7)

• Figure 1: This figure displays the percentage of CE (y-axis) that are required to predict a label for a single instance for 3 image datasets. Classes with more samples (majority) have lower class labels (e.g., 0), with the imbalance increasing from left to right, depending on exponential or step. A base model is trained with imbalanced data and 3 models are trained with over-sampling methods (x-axis). With a few exceptions, 12% or fewer CE are needed to predict a single instance of a minority or majority class. This holds whether the models are trained with or without minority class DA.
• Figure 2: This figure displays the percentage of CE (y-axis) required to predict a single instance for LG models on 5 tabular datasets. Class 0 (blue) denotes the majority and class 1 (brown) is the minority. A base model is trained on imbalanced data and 3 models are trained with over-sampling. For all datasets, more CE are required to predict minority classes, both with and without over-sampling.
• Figure 3: This figure displays the percentage of CE required to predict a single instance using SVM models for 5 UCI tabular datasets. Blue bars denote majority, and brown bars denote minority, classes. When training with base (imbalanced) data, majority class instances require, on average, almost all CE for prediction; however, when trained with SMOTE and ADASYN, and in some cases with REMIX, the percentage of CE falls to approx. 40% for both majority and minority classes.
• Figure 4: This figure compares the number of CE or features required to predict an instance versus the number of CE required to predict an entire class.
• Figure 5: This figure illustrates that, out of a total of 64 latent features in a Resnet-32 model trained to predict the CIFAR-10 truck class, only a handful of features have large feature magnitudes (y-axis). This figure depicts the minority class (class #9 - trucks). The indices of the latent features are shown on the x-axis.