Sparse Data-Driven Random Projection in Regression for High-Dimensional Data

TL;DR

The paper tackles high-dimensional linear regression with correlated predictors by proposing SPAR—a data-informed two-step approach that combines HOLP-based variable screening with a specially designed CW random projection. The key theoretical contribution is a bound showing substantial expected prediction error gains over conventional random projections when diagonal projection elements are aligned with the true coefficients. SPAR uses an ensemble of screened-projected models and a cross-validated sparsity threshold to balance prediction accuracy and sparsity, achieving strong performance across simulated covariance structures and sparsity levels. Empirical results on Rateye Gene Expression and Face Images demonstrate competitive predictive accuracy and interpretable coefficient patterns, highlighting SPAR’s practical utility for large, complex datasets. The work opens avenues for extensions to non-linear models and classification, while acknowledging sparsity-precision trade-offs in variable selection.

Abstract

We examine the linear regression problem in a challenging high-dimensional setting with correlated predictors where the vector of coefficients can vary from sparse to dense. In this setting, we propose a combination of probabilistic variable screening with random projection tools as a viable approach. More specifically, we introduce a new data-driven random projection tailored to the problem at hand and derive a theoretical bound on the gain in expected prediction error over conventional random projections. The variables to enter the projection are screened by accounting for predictor correlation. To reduce the dependence on fine-tuning choices, we aggregate over an ensemble of linear models. A thresholding parameter is introduced to obtain a higher degree of sparsity. Both this parameter and the number of models in the ensemble can be chosen by cross-validation. In extensive simulations, we compare the proposed method with other random projection tools and with classical sparse and dense methods and show that it is competitive in terms of prediction across a variety of scenarios with different sparsity and predictor covariance settings. We also show that the method with cross-validation is able to rank the variables satisfactorily. Finally, we showcase the method on two real data applications.

Figures (14)

• Figure 1: Comparison of screening based on marginal correlations, HOLP, Ridge with $\lambda =\sqrt{n} + \sqrt{p}$ and Ridge with cross-validated $\lambda$ in the setting in Example \ref{['ex:data_setting']}. (a) shows density estimates of absolute estimated coefficients for active and non-active predictors over $n_\text{rep}=100$ repetitions. (b) shows precision, recall, sign recovery, and correlation of estimates to the true coefficients averaged over $100$ replications, where the vertical line indicates the true number of active variables.
• Figure 2: Mean squared prediction errors of different conventional projections, the proposed projection using the HOLP coefficient (SparseCWHolp) or its signs (SparseCWSignH), and the oracle projections using the true $\beta$ (SparseCWBeta) or its signs (SparseCWSignB) for $n_\text{rep}=100$, $n=200, p=2000, m=a=100$ and $\Sigma=0.5\cdot 1_p1_p' + 0.5 \cdot I_p$.
• Figure 3: Average mean squared prediction error for using ensembles of screening using HOLP (Src_HOLP), CW random projections (RP_CW), screening with HOLP and conventional random projections (ScrRP_CW) and the combination of screening using HOLP and our proposed CW random projection (ScrRP) for different number of models for $n_\text{rep}=100$, $n=200, p=2000, m=a=100$ and $\Sigma=0.5\cdot 1_p1_p' + 0.5 \cdot I_p$.
• Figure 4: Average effect of number of screened variables on mean squared prediction error of only screening compared to screening plus random projection before fitting linear regression model for $100$ replications of the setting of Example \ref{['ex:data_setting']}.
• Figure 5: Relative MSPE of competing methods for different covariance and active predictor settings for $n_\text{rep}=100$ replications ($n=200,p=2000,\rho_\text{snr}=10$). Sparse methods are marked by dotted boxes.

Theorems & Definitions (14)

• Example 1
• Definition 1
• Theorem 1
• Remark 1
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4
• proof