Regularization Using Synthetic Data in High-Dimensional Models

TL;DR

The paper introduces the Synthetic-data Regularized Estimator (SRE) to regularize high-dimensional generalized linear models by augmenting the likelihood with weighted synthetic data from a simple model. Using the Convex Gaussian Min-Max Theorem, it derives exact asymptotic characterizations in the linear regime, including existence, stability, and minimax-rate results for non-separable regularization. It extends the theory to GLMs, provides practical inference tools (confidence intervals, calibration, and FDR-controlled variable selection), and demonstrates through simulations that SRE can outperform traditional regularization methods, especially when $p$ is large or the MLE fails to exist. The work further shows how informative auxiliary data can improve estimation and constructs tunable, theory-guided procedures for parameter selection, inference, and transfer learning in high-dimensional settings.

Abstract

To address the challenges of reliable statistical inference in high-dimensional models, we introduce the Synthetic-data Regularized Estimator (SRE). Unlike traditional regularization methods, the SRE regularizes the complex target model via a weighted likelihood based on synthetic data generated from a simpler, more stable model. This method provides a theoretically sound and practically effective alternative to parameter penalization. We establish key theoretical properties of the SRE in generalized linear models, including existence, stability, consistency, and minimax rate optimality. Applying the Convex Gaussian Min-Max Theorem, we derive a precise asymptotic characterization in the high-dimensional linear regime. To deal with the non-separable regularization, we introduce a novel decomposition in our analysis. Building upon these results, we develop practical methodologies for tuning parameter selection, confidence interval construction, and calibrated variable selection in high-dimensional inference. The effectiveness of the SRE is demonstrated through simulation studies and real-data applications.

Figures (16)

• Figure 1: An example of SREs in logistic regression using $\widehat{\boldsymbol{\beta}}_{M}$ with varying $\tau$ ($n=200$ and $p=250$). Observed data: ${\boldsymbol X}_i\sim N(\mathbf{0},\mathbf{I}_p)$ and $Y_i\sim \text{Bern}(\rho^\prime({\boldsymbol X}_i^\top {\boldsymbol \beta}_0))$ with $\|{\boldsymbol \beta}_0\|_2=2.5$. Synthetic data: ${\boldsymbol X}_i^*\sim N(\mathbf{0},\mathbf{I}_p)$, $Y_i^*\sim \text{Bern}(0.5)$, and $M=20p$.
• Figure 2: Performance of the SRE with non-informative synthetic data as a function of $\tau_0=\tau/n$. Each point is obtained by calculating the performance metrics of the SRE averaging over 50 simulation replications. The solid lines represent the corresponding theoretical prediction.
• Figure 3: Performance of the SRE with informative auxiliary data $(\kappa_2=1,\xi=0.9)$ as a function of $\tau_0=\tau/n$. Each point is obtained by calculating the performance metrics of the SRE averaging over 50 simulation replications. The solid lines represent the corresponding theoretical prediction.
• Figure 4: Performance of the pSRE $\widehat{{\boldsymbol \beta}}_\infty$ as a function of $\tau_0=\tau/n$. Each point is obtained by calculating the performance metrics of the pSRE averaging over 50 simulation replications. The solid lines represent the prediction by Conjecture \ref{['conjecture:exact_cat_Minfty_MAP_noninformative']}.
• Figure 5: Relationship between $\eta^2_{M}$ and $\kappa_1$ across different values of $\delta$. For each $\delta$, $\eta^2_M$ is computed using a grid of $\kappa_1$ values, with $\tau_0 = 1/4$ and $m = 20/\delta$.

Theorems & Definitions (71)

• Remark 1
• Remark 2
• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• Remark 3
• Theorem 3.4
• Corollary 3.5
• Theorem 4.1
• Corollary 4.2