Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Theoretical Convergence of SMOTE-Generated Samples

Firuz Kamalov, Hana Sulieman, Witold Pedrycz

TL;DR

The paper provides a rigorous theoretical analysis of SMOTE-generated samples. It proves that the synthetic variable $Z$ converges in probability to the original distribution $X$ as $n \to \infty$, and that $Z$ converges in mean when $X$ has compact support. It also shows that smaller neighbor rank $k$ speeds convergence, with empirical validation across uniform, Gaussian, and exponential distributions. The work offers a foundational understanding of SMOTE and guidance for parameter selection, with implications for broader data augmentation techniques.

Abstract

Imbalanced data affects a wide range of machine learning applications, from healthcare to network security. As SMOTE is one of the most popular approaches to addressing this issue, it is imperative to validate it not only empirically but also theoretically. In this paper, we provide a rigorous theoretical analysis of SMOTE's convergence properties. Concretely, we prove that the synthetic random variable Z converges in probability to the underlying random variable X. We further prove a stronger convergence in mean when X is compact. Finally, we show that lower values of the nearest neighbor rank lead to faster convergence offering actionable guidance to practitioners. The theoretical results are supported by numerical experiments using both real-life and synthetic data. Our work provides a foundational understanding that enhances data augmentation techniques beyond imbalanced data scenarios.

Theoretical Convergence of SMOTE-Generated Samples

TL;DR

The paper provides a rigorous theoretical analysis of SMOTE-generated samples. It proves that the synthetic variable converges in probability to the original distribution as , and that converges in mean when has compact support. It also shows that smaller neighbor rank speeds convergence, with empirical validation across uniform, Gaussian, and exponential distributions. The work offers a foundational understanding of SMOTE and guidance for parameter selection, with implications for broader data augmentation techniques.

Abstract

Imbalanced data affects a wide range of machine learning applications, from healthcare to network security. As SMOTE is one of the most popular approaches to addressing this issue, it is imperative to validate it not only empirically but also theoretically. In this paper, we provide a rigorous theoretical analysis of SMOTE's convergence properties. Concretely, we prove that the synthetic random variable Z converges in probability to the underlying random variable X. We further prove a stronger convergence in mean when X is compact. Finally, we show that lower values of the nearest neighbor rank lead to faster convergence offering actionable guidance to practitioners. The theoretical results are supported by numerical experiments using both real-life and synthetic data. Our work provides a foundational understanding that enhances data augmentation techniques beyond imbalanced data scenarios.
Paper Structure (6 sections, 7 theorems, 46 equations, 6 figures, 2 algorithms)

This paper contains 6 sections, 7 theorems, 46 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Main Results
  4. General SMOTE
  5. Optimal rank $k$
  6. Conclusion

Key Result

Theorem 3.1

Let $X$ be a continuous random variable. Let $Z$ be the random variable generated via the SMOTE-k procedure from an i.i.d. sample $X_1, X_2, \dots, X_n$ drawn from $X$. Then, $Z$ converges to $X$ in probability as $n\rightarrow \infty$.

Figures (6)

  • Figure 1: Convergence of SMOTE-generated samples $Z$ to the true population $X$ (UCI Air Quality CO data) as measured by the KS statistic. The downward trend confirms convergence in probability (Theorem \ref{['convProb']}). Lower values of $k$ result in faster convergence.
  • Figure 2: Convergence in mean of SMOTE-generated samples $Z$ to the true population $X$ (California Housing Median Income) as measured by the Wasserstein distance. The decreasing trend empirically validates Theorem \ref{['thm:convmean']}.
  • Figure 3: Simulated distribution of $Z$ for sample sizes $n = 8, 20, 70$ based on $X\sim U(0,1)$ together with the graph of $X$. It is clear that the distribution generated with $k=1$ (top) is closer to the true distribution than with $k=5$ (bottom).
  • Figure 4: Simulated distribution of $Z$ for sample sizes $n=8, 20, 70$ based on $X \sim \mathcal{N}(0,1)$ together with the graph of $X$. It is clear that the distribution generated with $k=1$ is closer to the true distribution than with $k=5$.
  • Figure 5: Simulated distribution of $Z$ for sample sizes $n=8, 20, 70$ based on $X \sim \mathrm{Exp}(1)$ together with the graph of $X$. It is clear that the distribution generated with $k=1$ is closer to the true distribution than with $k=5$.
  • ...and 1 more figures

Theorems & Definitions (13)

  • Theorem 3.1
  • proof
  • Lemma 1
  • Theorem 3.2
  • proof
  • Corollary 1
  • proof
  • Theorem 3.3
  • proof
  • Corollary 2
  • ...and 3 more