Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

TL;DR

The paper tackles sparse linear regression in high dimensions by introducing a generalized elastic-net model that uses an $\ell_r$ loss to accommodate different noise types and an $\ell_q$ penalty (0<q<1) to promote sparsity. To overcome the nonconvex, non-Lipschitz nature of the $\ell_q$ term, an $\epsilon$-approximation yields a convex, nonsmooth surrogate, and the authors prove that local minimizers are generalized first-order stationary points with explicit lower bounds on nonzero coefficients. They develop two efficient algorithms within an iterative reweighted framework: a straightforward ADMM and a faster PMM-SSN approach that solves the dual problem via semismooth Newton iterations. Theoretical results characterize stationarity and nonzero entry magnitude, while extensive simulations and real-data experiments show the PMM-SSN method generally outperforms ADMM and competitive state-of-the-art methods like IAGENR-Lq in accuracy and speed, across various noise regimes. Overall, the work provides a robust, scalable framework for sparse regression under general noise, with practical algorithms that yield high-precision solutions in high-dimensional settings.

Abstract

The elastic net penalty is frequently employed in high-dimensional statistics for parameter regression and variable selection. It is particularly beneficial compared to lasso when the number of predictors greatly surpasses the number of observations. However, empirical evidence has shown that the $\ell_q$-norm penalty (where $0 < q < 1$) often provides better regression compared to the $\ell_1$-norm penalty, demonstrating enhanced robustness in various scenarios. In this paper, we explore a generalized elastic net model that employs a $\ell_r$-norm (where $r \geq 1$) in loss function to accommodate various types of noise, and employs a $\ell_q$-norm (where $0 < q < 1$) to replace the $\ell_1$-norm in elastic net penalty. Theoretically, we establish the computable lower bounds for the nonzero entries of the generalized first-order stationary points of the proposed generalized elastic net model. For implementation, we develop two efficient algorithms based on the locally Lipschitz continuous $ε$-approximation to $\ell_q$-norm. The first algorithm employs an alternating direction method of multipliers (ADMM), while the second utilizes a proximal majorization-minimization method (PMM), where the subproblems are addressed using the semismooth Newton method (SNN). We also perform extensive numerical experiments with both simulated and real data, showing that both algorithms demonstrate superior performance. Notably, the PMM-SSN is efficient than ADMM, even though the latter provides a simpler implementation.

Figures (2)

• Figure 1: Distribution of predicted points at nonzero input locations
• Figure 2: RE values under different $q$ values and sparsity levels

Theorems & Definitions (14)

• Definition 1
• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Corollary 2.1
• Theorem 4
• proof