A statistical theory of overfitting for imbalanced classification

TL;DR

This work develops a statistical theory for high-dimensional imbalanced classification, focusing on linear classifiers such as SVM and logistic regression under a two-component Gaussian mixture model. It reveals that, in high dimensions, the empirical logit distribution on training data undergoes a truncation/rectification (a rectified Gaussian effect) that causes severe overfitting for the minority class, while the test logit distribution remains closer to a Gaussian. Margin rebalancing is shown to be essential for mitigating the minority accuracy drop, with explicit analyses of optimal margin ratios in proportional and high-imbalance regimes. The paper also connects overfitting to calibration metrics, providing monotone relationships and showing how optimal transport maps explain miscalibration, thus offering practical guidance for improving minority performance and uncertainty quantification in imbalanced, high-dimensional settings.

Abstract

Classification with imbalanced data is a common challenge in data analysis, where certain classes (minority classes) account for a small fraction of the training data compared with other classes (majority classes). Classical statistical theory based on large-sample asymptotics and finite-sample corrections is often ineffective for high-dimensional data, leaving many overfitting phenomena in empirical machine learning unexplained. In this paper, we develop a statistical theory for high-dimensional imbalanced classification by investigating support vector machines and logistic regression. We find that dimensionality induces truncation or skewing effects on the logit distribution, which we characterize via a variational problem under high-dimensional asymptotics. In particular, for linearly separable data generated from a two-component Gaussian mixture model, the logits from each class follow a normal distribution $\mathsf{N}(0,1)$ on the testing set, but asymptotically follow a rectified normal distribution $\max\{κ, \mathsf{N}(0,1)\}$ on the training set -- which is a pervasive phenomenon we verified on tabular data, image data, and text data. This phenomenon explains why the minority class is more severely affected by overfitting. Further, we show that margin rebalancing, which incorporates class sizes into the loss function, is crucial for mitigating the accuracy drop for the minority class. Our theory also provides insights into the effects of overfitting on calibration and other uncertain quantification measures.

Figures (14)

• Figure 1: Empirical logit distribution (ELD) and testing logit distribution (TLD). We train a max-margin classifier (namely SVM) $\widehat{f}$ on synthetic data from a 2-component Gaussian mixture model. Colors indicate labels $y_i$ and $x$-axis indicates logits $\widehat{f}({\boldsymbol{x}}_i)$. ELD for both classes: rectified Gaussian distribution (histogram). TLD for both classes: Gaussian distribution (curve). Overfitting effect: The density areas below the dotted curves are overlapping in TLD, thus leading to positive test error; however they are "pushed" to respective margin boundaries in ELD, thus leading to linear separability and zero training errors.
• Figure 2: ELD and TLD of logistic regression classifier (the last fully-connected layer) for real data. Left: IFNB single-cell RNA-seq dataset (tabular data). Middle: CIFAR-10 dataset preprocessed by pretrained ResNet-18 model for feature extraction (image data). Right: IMDb movie review dataset preprocessed by BERT base model (110M) for feature extraction (text data).
• Figure 3: Schematic illustration of margin-rebalanced SVM. The dotted line is the decision boundary for the original SVM, and the solid line is the decision boundary for margin-rebalanced SVM.
• Figure 4: Impact of imbalance on test errors. We show test errors from 2-GMM simulations with margin rebalancing (solid curves) and without (dashed curves) at three levels of signal strength $\| {\boldsymbol{\mu}}\|_2$ under varying imbalanced ratios $\pi$.
• Figure 5: Effects of margin rebalancing on test errors. We show test errors from 2-GMM simulations at three different imbalance ratios under varying $\tau$.

Theorems & Definitions (127)

• Definition 1.1: Logit and margin
• Definition 1.2: ELD and TLD
• Definition 1.3: Rectified Gaussian distribution
• Definition 1.4
• Theorem 1.1: Separable data, informal version of \ref{['thm:SVM_main']}
• Proposition 1.2: Optimal $\tau$ in proportional regime, informal version of \ref{['prop:tau_optimal']}
• Theorem 1.3: High imbalance
• Proposition 3.1
• Remark 3.1
• Proposition 3.2