Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry of Sparsity-Inducing Norms

Jean-Philippe Chancelier, Michel de Lara, Antoine Deza, Lionel Pournin

TL;DR

The paper addresses finding solutions with a priori sparsity budget $k$ by studying generalized $k$-support norms built from a source norm. It develops a geometric-dual framework that connects exposed faces of convex sets generated by $k$-sparse vectors to sparsity in optimal solutions, establishing dual conditions under which minimizers are $k$-sparse. It provides a detailed face analysis for unit balls of generalized $k$-support norms, including orthant-monotonic and orthant-strictly monotonic cases, and offers a complete geometric description of top-$(q,k)$ and $(p,k)$-support norms, especially in the $p=\infty$ and $1<p<\infty$ regimes. The results illuminate how dual information identifies support, and reveal the intricate face and normal-cone structure that underpins sparsity-inducing regularization. Collectively, the work advances the geometric understanding of sparsity-promoting norms and informs the design of penalties that enforce a fixed sparsity budget in optimization.

Abstract

Sparse optimization seeks an optimal solution with few nonzero entries. To achieve this, it is common to add to the criterion a penalty term proportional to the $\ell_1$-norm, which is recognized as the archetype of sparsity-inducing norms. In this approach, the number of nonzero entries is not controlled a priori. By contrast, in this paper, we focus on finding an optimal solution with at most~$k$ nonzero coordinates (or for short, $k$-sparse vectors), where $k$ is a given sparsity level (or ``sparsity budget''). For this purpose, we study the class of generalized $k$-support norms that arise from a given source norm. When added as a penalty term, we provide conditions under which such generalized $k$-support norms promote $k$-sparse solutions. The result follows from an analysis of the exposed faces of closed convex sets generated by $k$-sparse vectors, and of how primal support identification can be deduced from dual information. Finally, we study some of the geometric properties of the unit balls for the $k$-support norms and their dual norms when the source norm belongs to the family of $\ell_p$-norms.

Geometry of Sparsity-Inducing Norms

TL;DR

The paper addresses finding solutions with a priori sparsity budget by studying generalized -support norms built from a source norm. It develops a geometric-dual framework that connects exposed faces of convex sets generated by -sparse vectors to sparsity in optimal solutions, establishing dual conditions under which minimizers are -sparse. It provides a detailed face analysis for unit balls of generalized -support norms, including orthant-monotonic and orthant-strictly monotonic cases, and offers a complete geometric description of top- and -support norms, especially in the and regimes. The results illuminate how dual information identifies support, and reveal the intricate face and normal-cone structure that underpins sparsity-inducing regularization. Collectively, the work advances the geometric understanding of sparsity-promoting norms and informs the design of penalties that enforce a fixed sparsity budget in optimization.

Abstract

Sparse optimization seeks an optimal solution with few nonzero entries. To achieve this, it is common to add to the criterion a penalty term proportional to the -norm, which is recognized as the archetype of sparsity-inducing norms. In this approach, the number of nonzero entries is not controlled a priori. By contrast, in this paper, we focus on finding an optimal solution with at most~ nonzero coordinates (or for short, -sparse vectors), where is a given sparsity level (or ``sparsity budget''). For this purpose, we study the class of generalized -support norms that arise from a given source norm. When added as a penalty term, we provide conditions under which such generalized -support norms promote -sparse solutions. The result follows from an analysis of the exposed faces of closed convex sets generated by -sparse vectors, and of how primal support identification can be deduced from dual information. Finally, we study some of the geometric properties of the unit balls for the -support norms and their dual norms when the source norm belongs to the family of -norms.
Paper Structure (17 sections, 15 theorems, 89 equations, 6 figures, 1 table)

This paper contains 17 sections, 15 theorems, 89 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Face characterization and support identification
  3. Background on sparsity and on faces of closed convex sets
  4. Convex sets with $k$-sparse extreme points
  5. Proofs of Theorem \ref{['th:Support_identification']} and Corollary \ref{['cor:Support_identification']}
  6. Proof of Theorem \ref{['th:Support_identification']}
  7. Proof of Corollary \ref{['cor:Support_identification']}
  8. The case of generalized top-$k$ and $k$-support norms
  9. Background on generalized top-$k$ and $k$-support norms
  10. Face characterization and support identification
  11. The orthant-monotonic case
  12. The orthant-strictly monotonic case
  13. Geometry of the top-($q$,$k$) and ($p$,$k$)-support norms
  14. The top-($q$,$k$) norm and ($p$,$k$)-support norms
  15. The case when $p$ is equal to $+\infty$
  16. ...and 2 more sections

Key Result

Theorem 1

Let $k\in\llbracket 1,{d} \rrbracket$ be a natural number and $X \subset {\mathbb R}^{d}$ be a (primal) nonempty set. We set Let $y \in {\mathbb R}^{d}$ be a (dual) vector. We set which is such that $\emptyset \neq {\mathcal{K}}^\sharp_{X,k}({y}) \subset \{{ K \subset\llbracket 1,{d} \rrbracket, {\lvert K \rvert}\leq k}\} \subset 2^{\llbracket 1,{d} \rrbracket}$. Then, we have that the set $X_{

Figures (6)

  • Figure 1: Replica of Tibshirani:1996
  • Figure 2: Two examples of unit balls with kinks located at sparse points
  • Figure 3: Unit balls ${B}_{\infty,1}^{\top\!\star}$ (left) and ${B}_{1,1}^{\top}$ (right) when $d=3$
  • Figure 4: Unit balls ${B}_{\infty,2}^{\top\!\star}$ (left) and ${B}_{1,2}^{\top}$ (right) when $d=3$
  • Figure 5: Unit balls ${B}_{2,2}^{\top\!\star}$ (left) and ${B}_{2,2}^{\top}$ (right) when $d=3$
  • ...and 1 more figures

Theorems & Definitions (16)

  • Theorem 1: Characterization of faces
  • Corollary 2: Support identification
  • Lemma 3
  • Definition 4
  • Proposition 5
  • Theorem 6
  • Corollary 7
  • Proposition 8
  • Proposition 9
  • Theorem 10
  • ...and 6 more