A Theoretical Analysis of the Learning Dynamics under Class Imbalance

TL;DR

The paper addresses learning dynamics under class imbalance and shows that gradient-based training can exhibit a minority initial drop (MID) when gradients from the majority class dominate. It introduces Per-Class Normalized Gradient Descent (PCNGD) and Per-Class Normalized SGD (PCNSGD) to equalize per-class gradient signals, deriving convergence guarantees for GD and highlighting the need for additional stabilization (oversampling, rescaling) in SGD due to directional noise. The authors provide theoretical results under smoothness and gradient-dominance conditions, proving that PCNGD can monotonically decrease per-class losses and, in gradient-dominated settings, converge to per-class minima; in SGD, variance and directional noise necessitate PCNSGD+O and PCNSGD+R to achieve robust, balanced learning. Empirical validation across multiple architectures and long-tailed CIFAR-derived datasets demonstrates MID elimination, faster per-class convergence, and improved generalization with PCN variants, while also explaining when oversampling is particularly beneficial. Overall, the work offers a principled framework for addressing class imbalance by combining per-class gradient normalization with conventional data- and loss-level strategies, providing both convergence guarantees and practical guidance for real-world long-tailed problems.

Abstract

Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.

Figures (11)

• Figure 1: Training (left) and test (right) recall of the Mod1 on the Bi60 datasets (see details in Sec. \ref{['sec:empirical_PCNGD']}). Due to the data imbalance, GD (blue curves) first focuses on the majority class only, while the minority class stays at zero accuracy. This effect is suppressed when using the PCNGD algorithm. See Fig. \ref{['fig:loss60']} for the related loss function curves.
• Figure 2: Diagram explaining the directional noise caused by class imbalance on (PCN)SGD in binary classification. We plot two generic components of the parameter vector ${\bf x}$ on the axes ($\mathrm{x^i}$, $\mathrm{x^j}$). Starting from a given iterate ${\bf x}_t$ at time $t$, the normalized per-class gradients associated with individual batches, and the entire dataset are represented in the plots. The gradients of the individual examples that make them up are also shown. (a) The contribution of each example to the per-class full-batch gradient (FBG). (b) Within a mini batch, instead, we consider only a randomly selected subset of dataset elements; the mini-batch gradient will therefore come from a random selection in this subset. (c) We show several mini batches and observe that they are more aligned to the FBG of the majority class than the FBG of the minority class. We will see this has negative consequences on the dynamics. For more details, see Sec. \ref{['sec:directional-noise']}.
• Figure 3: Training (left) and test (right) loss curves for GD and PCNGD algorithms. The corresponding recall is shown for the same setup in \ref{['fig:warmup']}.
• Figure 4: Per-class train recall of PCNSGD and of PCNSGD+R with Mod1 on the Bi7a dataset. With the rescaling proposed in Eq. \ref{['eq:rescaling']}, leading to PCNSGD+R, the MID disappears. The macro-averaged curves are shown in Fig. \ref{['fig:PCNSGD']}.
• Figure 5: Comparison between different stochastic algorithms with model Mod1 on the Bi7a dataset.

Theorems & Definitions (23)

• Theorem 4.1: Informal
• Theorem 4.2: Informal
• proof : Proof sketch for $f^{(0)}$
• Theorem 4.3
• Theorem 5.1: informal
• Theorem 3.1: Formal version of Theorem \ref{['th:GDdecreasing']}
• proof
• Theorem 3.2
• proof
• Theorem 4.1: Formal version of Theorem \ref{['th:PCNGDdecreasing']}