Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Piecewise Linearity of Min-Norm Solution Map of a Nonconvexly Regularized Convex Sparse Model

Yi Zhang, Isao Yamada

TL;DR

An extension of the least angle regression (LARS) algorithm, which iteratively computes the closed-form expression of $\mathbf{x}_{\star}(\mathbf{y},\lambda)$ in each linear zone, and provably obtains the whole solution map $\mathbf{x}_{\star}(\mathbf{y,\lambda)$ within finite iterations.

Abstract

It is well known that the minimum $\ell_2$-norm solution of the convex LASSO model, say $\mathbf{x}_{\star}$, is a continuous piecewise linear function of the regularization parameter $λ$, and its signed sparsity pattern is constant within each linear piece. The current study is an extension of this classic result, proving that the aforementioned properties extend to the min-norm solution map $\mathbf{x}_{\star}(\mathbf{y},λ)$, where $\mathbf{y}$ is the observed signal, for a generalization of LASSO termed the scaled generalized minimax concave (sGMC) model. The sGMC model adopts a nonconvex debiased variant of the $\ell_1$-norm as sparse regularizer, but its objective function is overall-convex. Based on the geometric properties of $\mathbf{x}_{\star}(\mathbf{y},λ)$, we propose an extension of the least angle regression (LARS) algorithm, which iteratively computes the closed-form expression of $\mathbf{x}_{\star}(\mathbf{y},λ)$ in each linear zone. Under suitable conditions, the proposed algorithm provably obtains the whole solution map $\mathbf{x}_{\star}(\mathbf{y},λ)$ within finite iterations. Notably, our proof techniques for establishing continuity and piecewise linearity of $\mathbf{x}_{\star}(\mathbf{y},λ)$ are novel, and they lead to two side contributions: (a) our proofs establish continuity of the sGMC solution set as a set-valued mapping of $(\mathbf{y},λ)$; (b) to prove piecewise linearity and piecewise constant sparsity pattern of $\mathbf{x}_{\star}(\mathbf{y},λ)$, we do not require any assumption that previous work relies on (whereas to prove some additional properties of $\mathbf{x}_{\star}(\mathbf{y},λ)$, we use a different set of assumptions from previous work).

Piecewise Linearity of Min-Norm Solution Map of a Nonconvexly Regularized Convex Sparse Model

TL;DR

An extension of the least angle regression (LARS) algorithm, which iteratively computes the closed-form expression of in each linear zone, and provably obtains the whole solution map within finite iterations.

Abstract

It is well known that the minimum -norm solution of the convex LASSO model, say , is a continuous piecewise linear function of the regularization parameter , and its signed sparsity pattern is constant within each linear piece. The current study is an extension of this classic result, proving that the aforementioned properties extend to the min-norm solution map , where is the observed signal, for a generalization of LASSO termed the scaled generalized minimax concave (sGMC) model. The sGMC model adopts a nonconvex debiased variant of the -norm as sparse regularizer, but its objective function is overall-convex. Based on the geometric properties of , we propose an extension of the least angle regression (LARS) algorithm, which iteratively computes the closed-form expression of in each linear zone. Under suitable conditions, the proposed algorithm provably obtains the whole solution map within finite iterations. Notably, our proof techniques for establishing continuity and piecewise linearity of are novel, and they lead to two side contributions: (a) our proofs establish continuity of the sGMC solution set as a set-valued mapping of ; (b) to prove piecewise linearity and piecewise constant sparsity pattern of , we do not require any assumption that previous work relies on (whereas to prove some additional properties of , we use a different set of assumptions from previous work).
Paper Structure (48 sections, 20 theorems, 178 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 48 sections, 20 theorems, 178 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Works
  4. Paper Organization
  5. Notations
  6. Numbers, Sets and Functions
  7. Vectors and Matrices
  8. Static Solution-Set Geometry
  9. Three Equivalent Formulations of the sGMC Model and the Optimality Condition
  10. Shape, Uniqueness and Sparseness of the Solution Sets
  11. Min-Norm Solution Map Analysis
  12. Continuity Properties of the Solution Sets, Min-Norm Solution Maps and Associated Quantities
  13. Preparation for Studying Piecewise Simple Behavior of the Min-Norm Solution Map
  14. Piecewise Simple Behavior of the Min-Norm Solution Map and Continuous-Discrete Correspondence
  15. Extended Least Angle Regression
  16. ...and 33 more sections

Key Result

Lemma 1

Given $(\bm{A},\rho,\bm{b},\lambda)\in\mathscr{P}$, $\bm{w}\in\mathbb{R}^{2n}$ is an extended solution in $\mathcal{S}_{\text{e}}$ (i.e., $\bm{w}$ is the concatenation of some primal solution $\bm{x}\in\mathcal{S}_{\text{p}}$ and dual solution $\bm{z}\in\mathcal{S}_{\text{d}}$) if and only if the fo where $\bm{C} \coloneqq \mathrm{blkdiag}(\bm{A},\sqrt{\rho} \bm{A})$, and $\bm{c}_i$ is the $i$th

Figures (1)

  • Figure : A single iteration of the E-LARS algorithm

Theorems & Definitions (38)

  • Definition 1: dossal2012
  • Definition 2
  • Lemma 1
  • Theorem 1
  • Remark 1: Consequences of Theorem \ref{['thm:shape_uniqueness_sparseness']}
  • Remark 2: Proof sketch of Theorem \ref{['thm:shape_uniqueness_sparseness']}
  • Corollary 1
  • Definition 3: rockafellar2009
  • Definition 4: rockafellar2009
  • Theorem 2
  • ...and 28 more