Admissibility Breakdown in High-Dimensional Sparse Regression with L1 Regularization
Guo Liu
TL;DR
This paper investigates tuning the Lasso in high-dimensional sparse regression and shows that the conventional non-asymptotic tuning $\lambda \sim \sigma\sqrt{\log(p)/n}$ can be inadmissible with respect to mean squared prediction error. By constructing a Lasso–Ridge refinement and decomposing the risk via a DMSE framework, the authors derive explicit lower bounds that demonstrate prediction improvements over a broad range of $\lambda_L$, provided the second-stage penalty uses $\lambda_R = c|E|$ with $c>2$. The analysis leverages a detailed KKT-based decomposition, Gaussian maxima bounds, and projections onto the active subspace, revealing how design structure and approximate support containment drive inadmissibility. Extensions to Dantzig selector and Adaptive Lasso indicate the approach's broader relevance to $\ell_1$-based methods, offering a theoretical explanation for empirical gains from smaller tuning in correlated designs and suggesting practical refinements for non-asymptotic regimes.
Abstract
The choice of the tuning parameter in the Lasso is central to its statistical performance in high-dimensional linear regression. Classical consistency theory identifies the rate of the Lasso tuning parameter, and numerous studies have established non-asymptotic guarantees. Nevertheless, the question of optimal tuning within a non-asymptotic framework has not yet been fully resolved. We establish tuning criteria above which the Lasso becomes inadmissible under mean squared prediction error. More specifically, we establish thresholds showing that certain classical tuning choices yield Lasso estimators strictly dominated by a simple Lasso-Ridge refinement. We also address how the structure of the design matrix and the noise vector influences the inadmissibility phenomenon.