Sparse learning with concave regularization: relaxation of the irrepresentable condition

TL;DR

This work addresses sparse recovery under compressed sensing by criticizing Lasso's reliance on the irrepresentable condition (IRR) for correct support recovery. It introduces MCPS$^2$, a concave MCP-based regularization with the cost $\mathcal{F}(x)=\tfrac12\|y-Ax\|_2^2+\lambda r(x)$ and $r(x)=d\|x\|_1-\tfrac12\|x\|_2^2$, and shows that variable-selection consistency can hold under a relaxed IRR, aided by restricted eigenvalue (RE) properties. The paper analyzes local and global minima of MCPS$^2$, including noise-free and binary-valued cases, and provides conditions under which the true sparse vector is recovered. It also develops algorithms to compute the minimum, including semi-definite relaxations (SDR) and ADMM, and reports numerical results where MCPS$^2$ achieves higher VSC than Lasso with fewer measurements. Overall, the approach offers a practical path to accurate sparse recovery with reduced data requirements, supported by theoretical relaxations and algorithmic strategies.

Abstract

Learning sparse models from data is an important task in all those frameworks where relevant information should be identified within a large dataset. This can be achieved by formulating and solving suitable sparsity promoting optimization problems. As to linear regression models, Lasso is the most popular convex approach, based on an $\ell_1$-norm regularization. In contrast, in this paper, we analyse a concave regularized approach, and we prove that it relaxes the irrepresentable condition, which is sufficient and essentially necessary for Lasso to select the right significant parameters. In practice, this has the benefit of reducing the number of necessary measurements with respect to Lasso. Since the proposed problem is non-convex, we also discuss different algorithms to solve it, and we illustrate the obtained enhancement via numerical experiments.

Figures (3)

• Figure 1: Irrepresentable condition rate \ref{['lasso_irr']} (Lasso) and C3 (MCPS$^2$), over 1000 runs, for $\eta=0$ (top) and $\|\eta\|_{\infty}=10^{-3}$ (bottom). In both cases, the irrepresentable condition that guarantees the VSC of MCPS$^2$ is satisfied more often than that of Lasso, which encourage the use of MCPS$^2$.
• Figure 2: Comparison between the proposed MCPS$^2$ optimization problem, solved via SDR and ADMM, and the Lasso, solved via ADMM in terms of VSC
• Figure 3: Comparison between the proposed MCPS$^2$ optimization problem, solved via SDR and ADMM, and the Lasso, solved via ADMM: convergence time

Theorems & Definitions (12)

• Definition 1
• Definition 2
• Lemma 1
• proof
• Theorem 1
• proof
• Remark 1
• Remark 2
• Remark 3
• Proposition 1