Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Classification Imbalance as Transfer Learning

Eric Xia, Jason M. Klusowski

TL;DR

This work reframes binary class imbalance as a label-shift transfer-learning problem, aiming to learn a classifier that performs well under a balanced target distribution $\mathbb{Q}$ when data come from a skewed source $\mathbb{P}$. It develops a general rebalancing framework that augments the minority class with synthetic samples drawn from a generator $\widehat{\mathbb{P}}_{X|Y=1}$ and derives excess-risk bounds decomposed into a target-oracle term plus a transfer-cost term tied to distributional discrepancy, notably $d_{TV}$ and $\chi^2$-divergences. The paper provides fast-rate bounds via localization, showing that under Lipschitz/strong convexity, the estimation error can achieve near-parametric rates with explicit transfer-cost penalties, particularly revealing a dimension-dependent $N_1^{-1/d}$ term for SMOTE that motivates preferring bootstrapping in higher dimensions. Concrete instantiations (kernel density estimation, diffusion sampling, bootstrapping, SMOTE) yield explicit nonasymptotic guarantees and practical guidance, including recommending bootstrapping over SMOTE in most high-dimensional regimes and suggesting density or diffusion-based strategies when structure or scores are available. Overall, the theory offers a unified, transfer-learning-based lens on imbalance, delivering actionable guidance for selecting augmentation strategies and establishing principled trade-offs between oracle performance and transfer costs.

Abstract

Classification imbalance arises when one class is much rarer than the other. We frame this setting as transfer learning under label (prior) shift between an imbalanced source distribution induced by the observed data and a balanced target distribution under which performance is evaluated. Within this framework, we study a family of oversampling procedures that augment the training data by generating synthetic samples from an estimated minority-class distribution to roughly balance the classes, among which the celebrated SMOTE algorithm is a canonical example. We show that the excess risk decomposes into the rate achievable under balanced training (as if the data had been drawn from the balanced target distribution) and an additional term, the cost of transfer, which quantifies the discrepancy between the estimated and true minority-class distributions. In particular, we show that the cost of transfer for SMOTE dominates that of bootstrapping (random oversampling) in moderately high dimensions, suggesting that we should expect bootstrapping to have better performance than SMOTE in general. We corroborate these findings with experimental evidence. More broadly, our results provide guidance for choosing among augmentation strategies for imbalanced classification.

Classification Imbalance as Transfer Learning

TL;DR

This work reframes binary class imbalance as a label-shift transfer-learning problem, aiming to learn a classifier that performs well under a balanced target distribution when data come from a skewed source . It develops a general rebalancing framework that augments the minority class with synthetic samples drawn from a generator and derives excess-risk bounds decomposed into a target-oracle term plus a transfer-cost term tied to distributional discrepancy, notably and -divergences. The paper provides fast-rate bounds via localization, showing that under Lipschitz/strong convexity, the estimation error can achieve near-parametric rates with explicit transfer-cost penalties, particularly revealing a dimension-dependent term for SMOTE that motivates preferring bootstrapping in higher dimensions. Concrete instantiations (kernel density estimation, diffusion sampling, bootstrapping, SMOTE) yield explicit nonasymptotic guarantees and practical guidance, including recommending bootstrapping over SMOTE in most high-dimensional regimes and suggesting density or diffusion-based strategies when structure or scores are available. Overall, the theory offers a unified, transfer-learning-based lens on imbalance, delivering actionable guidance for selecting augmentation strategies and establishing principled trade-offs between oracle performance and transfer costs.

Abstract

Classification imbalance arises when one class is much rarer than the other. We frame this setting as transfer learning under label (prior) shift between an imbalanced source distribution induced by the observed data and a balanced target distribution under which performance is evaluated. Within this framework, we study a family of oversampling procedures that augment the training data by generating synthetic samples from an estimated minority-class distribution to roughly balance the classes, among which the celebrated SMOTE algorithm is a canonical example. We show that the excess risk decomposes into the rate achievable under balanced training (as if the data had been drawn from the balanced target distribution) and an additional term, the cost of transfer, which quantifies the discrepancy between the estimated and true minority-class distributions. In particular, we show that the cost of transfer for SMOTE dominates that of bootstrapping (random oversampling) in moderately high dimensions, suggesting that we should expect bootstrapping to have better performance than SMOTE in general. We corroborate these findings with experimental evidence. More broadly, our results provide guidance for choosing among augmentation strategies for imbalanced classification.
Paper Structure (69 sections, 29 theorems, 284 equations, 1 figure, 2 algorithms)

This paper contains 69 sections, 29 theorems, 284 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Problem setting
  4. SMOTE and bootstrapping
  5. Related work
  6. Our contributions
  7. Paper organization and notation
  8. Rebalancing approaches
  9. Some intuition
  10. Excess risk bounds
  11. Localization bounds
  12. Concrete instantiations
  13. Guarantees for specific $\widehat{\mathbb{P}}_{X \mid Y = 1}$
  14. Continuous $\widehat{\mathbb{P}}_{X \mid Y = 1}$
  15. Discrete $\widehat{\mathbb{P}}_{X \mid Y = 1}$
  16. ...and 54 more sections

Key Result

Theorem 1

Given observations $\mathbb{D} = \{(X_i, Y_i)\}_{i=1}^N \stackrel{i.i.d.}{\sim}\mathbb{P}_{X, Y}$, consider Algorithm AlgRebal implemented with an arbitrary synthetic sampling distribution $\widehat{\mathbb{P}}_{X \mid Y = 1}$ and a $b$-uniformly bounded loss eqn:bound-loss. Then for any $\delta \in

Figures (1)

  • Figure 1: Illustration of the ratio of the excess risk of SMOTE and bootstrapping. The $x$-axis shows the underlying dimension $d$, and the different colored lines represent different sample sizes. As we can see, for a fixed sample size, increasing $d$ worsens the relative performance of SMOTE compared to bootstrapping. For moderately large $d$, increasing $N$ will also worsen the relative performance.

Theorems & Definitions (29)

  • Theorem 1
  • Theorem 2
  • Proposition 1
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • ...and 19 more