A First Course in Sparse Optimization

TL;DR

The book addresses the challenge of extracting meaningful, low-dimensional structure from high-dimensional data by promoting sparsity. It combines two intertwined paradigms—compressed sensing for sparse signal recovery and LASSO-type regularization for sparse modeling—supported by convex relaxations such as basis pursuit and elastic-net. The text builds a rigorous toolkit: foundational linear algebra (norms, SVD, projections, PSD/PD), probabilistic tools (union bounds, MGFs, concentration inequalities), and properties of Gaussian models, all to establish guarantees and practical algorithms for sparse recovery and sparse learning. Its contributions include a unified, mathematically rigorous treatment of sparse optimization, detailed algorithmic approaches, and guidance on measurement design (RIP, random matrices) to enable exact or stable recovery in science and engineering.

Abstract

This article aims to provide a comprehensive overview of sparse optimization, with a focus on both sparse signal recovery and sparse regularization techniques. We will begin by exploring the foundations of sparse optimization, delving into the mathematical tools and models that underpin sparse signal recovery and LASSO. We will then discuss key algorithms for both sparse recovery (e.g., basis pursuit, matching pursuit) and sparse regularization (e.g., LASSO, elastic net), along with their applications in real-world problems. Throughout the text, we balance intuitive explanations with rigorous mathematical formulations to provide a comprehensive resource for both newcomers and experts in the field. Our aim is twofold: to provide a self-contained entry point for students and researchers new to the field, and to offer a rigorous reference for practitioners seeking to apply sparse optimization in science and engineering.

Figures (24)

• Figure 1: Projection onto the hyperplane $\mathcal{C}({\bm{X}})$, i.e., the column space of ${\bm{X}}$.
• Figure 2: Comparison between the reduced and full SVD. White entries are zero, and blue entries are not necessarily zero
• Figure 3: Orthonormal bases that diagonalize ${\bm{X}}$ via the SVD. The set $\{{\bm{v}}_1, {\bm{v}}_2, \ldots, {\bm{v}}_r\}$ forms an orthonormal basis for the row space $\mathcal{C}({\bm{X}}^\top)$, and $\{{\bm{u}}_1,{\bm{u}}_2, \ldots,{\bm{u}}_r\}$ forms an orthonormal basis for the column space $\mathcal{C}({\bm{X}})$. The action of ${\bm{X}}$ links these bases: for each $i \in \{1, 2, \ldots, r\}$, it transforms the row-space basis vector ${\bm{v}}_i$ into the column-space basis vector ${\bm{u}}_i$ scaled by the singular value $\sigma_i$, i.e., ${\bm{X}} {\bm{v}}_i = \sigma_i {\bm{u}}_i$.
• Figure 4: Column space and row space of the pseudo-inverse ${\bm{X}}^+$. ${\bm{X}}$ transfers from the row space to the column space. ${\bm{X}}^+$ maps from the column space to the row space.
• Figure 5: Density and contour plots (blue=low, yellow=high) of the multivariate Gaussian distribution over the $\mathbb{R}^2$ space for various values of the covariance/scale matrix with a zero-mean vector. Fig \ref{['fig:dists_multiGauss_sigma1']} and \ref{['fig:dists_multiGauss_sigma4']}: A spherical covariance matrix has a circular shape; Fig \ref{['fig:dists_multiGauss_sigma2']} and \ref{['fig:dists_multiGauss_sigma5']}: A diagonal covariance matrix yields axis-aligned elliptical contours; Fig \ref{['fig:dists_multiGauss_sigma3']} and \ref{['fig:dists_multiGauss_sigma6']}: A full covariance matrix results in rotated elliptical contours.

Theorems & Definitions (123)

• Definition 1.1: Matlab notation
• Definition 1.2: Eigenvalue, eigenvector
• Definition 1.4: Subspace and span
• Definition 1.5: Linearly independent
• Definition 1.6: Basis and dimension
• Definition 1.7: Column space (range) and row space
• Definition 1.8: Null space (nullspace, kernel)
• Definition 1.9: Rank
• Definition 1.10: Orthogonal complement in general
• Definition 1.11: Orthogonal complement of column space