A Generalized Theory of Mixup for Structure-Preserving Synthetic Data

TL;DR

This work analyzes the statistical properties of mixup-generated data, showing that standard mixup preserves the mean of continuous features but can distort variance and joint structure. It derives explicit conditions for preserving (co)variance and conditional moments, and introduces EpBeta, a four-parameter weight distribution that expands mixup support to $[-\epsilon_0,1+\epsilon_1]$ to bound distortion with a tunable $\delta$. The authors provide theoretical guarantees showing Var and Cov can be preserved under appropriate parameter choices satisfying equations involving $\alpha,\beta,\epsilon_0,\epsilon_1$, and demonstrate a practical parameter-selection workflow. Empirical results on tabular and image data show that EpBeta maintains key distributional properties and sustains model performance under repeated synthesis, addressing concerns about model collapse and performing comparably to existing synthetic-data methods.

Abstract

Mixup is a widely adopted data augmentation technique known for enhancing the generalization of machine learning models by interpolating between data points. Despite its success and popularity, limited attention has been given to understanding the statistical properties of the synthetic data it generates. In this paper, we delve into the theoretical underpinnings of mixup, specifically its effects on the statistical structure of synthesized data. We demonstrate that while mixup improves model performance, it can distort key statistical properties such as variance, potentially leading to unintended consequences in data synthesis. To address this, we propose a novel mixup method that incorporates a generalized and flexible weighting scheme, better preserving the original data's structure. Through theoretical developments, we provide conditions under which our proposed method maintains the (co)variance and distributional properties of the original dataset. Numerical experiments confirm that the new approach not only preserves the statistical characteristics of the original data but also sustains model performance across repeated synthesis, alleviating concerns of model collapse identified in previous research.

Figures (16)

• Figure 1: A contour plot of a sample comprising $1000$ data points from $N(\left(00\right), \left(1.10.90.91.1\right))$ is shown in the first plot. This original data is then synthesized with three different mixup weight distributions: the proposed $\operatorname{EpBeta}(4.34,1.33;0.3,0.3)$, as well as two standard distributions, $\operatorname{Beta}(0.1,0.1)$ and $\operatorname{Unif}(0, 1)$. While the synthetic data generated by $\operatorname{Beta}$ or $\operatorname{Unif}$ is shrunken, the proposed $\operatorname{EpBeta}$ preserves the original data structure.
• Figure 2: (Left) All $(\alpha,\beta)$ pairs that gives structure-preserving synthetic data for choices of $(\epsilon_0,\epsilon_1)$. (Right) Density functions corresponding to the solid circled point of each curve on the left plot.
• Figure 3: The relative bias of covariance (triangle) and expectation (bar at bottom) for the 'Abalone' data using various synthetic generation methods. Negative bias is colored in blue and positive bias in red. Grey represents bias close to zero.
• Figure 4: The estimated polynomial regression coefficient and its $95\%$ confidence interval for each synthetic data. The red horizontal line represents the coefficient estimate from the original data.
• Figure 5: The relative bias of (co)variance (triangle) and expectation (bar) for 'Abalone' with $m=n$.

Theorems & Definitions (41)

• Lemma 3.1: Variance
• Theorem 3.2: Covariance
• Corollary 3.3: Correlation
• Definition 1
• Theorem 3.4: Conditional Mean
• Corollary 3.5: Conditional Mean Gap
• Theorem 3.6: Conditional Variance Gap
• Lemma 3.7
• Lemma 3.8: Optimal Cut Point $\tau$
• Corollary 4.1: Variance-Reduction Mixup