Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The greedy side of the LASSO: New algorithms for weighted sparse recovery via loss function-based orthogonal matching pursuit

Sina Mohammad-Taheri, Simone Brugiapaglia

TL;DR

The paper tackles weighted sparse recovery by bridging greedy algorithms and convex relaxations through loss-function-based OMP. It develops WLASSO-, SR-LASSO-, and WLAD-LASSO-based OMP variants, deriving explicit loss-reduction formulas that allow greedy index selection and local data-fitting updates. The key contributions include explicit formulas for loss reductions, noise-robust tuning behavior for SR-LASSO, and fault tolerance for LAD-LASSO, with empirical evidence showing improved accuracy and runtime efficiency over standard OMP. The approach provides a principled way to incorporate prior information and different noise models into greedy sparse recovery, with strong implications for high-dimensional function approximation and compressive sensing tasks.

Abstract

We propose a class of greedy algorithms for weighted sparse recovery by considering new loss function-based generalizations of Orthogonal Matching Pursuit (OMP). Given a (regularized) loss function, the proposed algorithms alternate the iterative construction of the signal support via greedy index selection and a signal update based on solving a local data-fitting problem restricted to the current support. We show that greedy selection rules associated with popular weighted sparsity-promoting loss functions admit explicitly computable and simple formulas. Specifically, we consider $ \ell^0 $- and $ \ell^1 $-based versions of the weighted LASSO (Least Absolute Shrinkage and Selection Operator), the Square-Root LASSO (SR-LASSO) and the Least Absolute Deviations LASSO (LAD-LASSO). Through numerical experiments on Gaussian compressive sensing and high-dimensional function approximation, we demonstrate the effectiveness of the proposed algorithms and empirically show that they inherit desirable characteristics from the corresponding loss functions, such as SR-LASSO's noise-blind optimal parameter tuning and LAD-LASSO's fault tolerance. In doing so, our study sheds new light on the connection between greedy sparse recovery and convex relaxation.

The greedy side of the LASSO: New algorithms for weighted sparse recovery via loss function-based orthogonal matching pursuit

TL;DR

The paper tackles weighted sparse recovery by bridging greedy algorithms and convex relaxations through loss-function-based OMP. It develops WLASSO-, SR-LASSO-, and WLAD-LASSO-based OMP variants, deriving explicit loss-reduction formulas that allow greedy index selection and local data-fitting updates. The key contributions include explicit formulas for loss reductions, noise-robust tuning behavior for SR-LASSO, and fault tolerance for LAD-LASSO, with empirical evidence showing improved accuracy and runtime efficiency over standard OMP. The approach provides a principled way to incorporate prior information and different noise models into greedy sparse recovery, with strong implications for high-dimensional function approximation and compressive sensing tasks.

Abstract

We propose a class of greedy algorithms for weighted sparse recovery by considering new loss function-based generalizations of Orthogonal Matching Pursuit (OMP). Given a (regularized) loss function, the proposed algorithms alternate the iterative construction of the signal support via greedy index selection and a signal update based on solving a local data-fitting problem restricted to the current support. We show that greedy selection rules associated with popular weighted sparsity-promoting loss functions admit explicitly computable and simple formulas. Specifically, we consider - and -based versions of the weighted LASSO (Least Absolute Shrinkage and Selection Operator), the Square-Root LASSO (SR-LASSO) and the Least Absolute Deviations LASSO (LAD-LASSO). Through numerical experiments on Gaussian compressive sensing and high-dimensional function approximation, we demonstrate the effectiveness of the proposed algorithms and empirically show that they inherit desirable characteristics from the corresponding loss functions, such as SR-LASSO's noise-blind optimal parameter tuning and LAD-LASSO's fault tolerance. In doing so, our study sheds new light on the connection between greedy sparse recovery and convex relaxation.
Paper Structure (52 sections, 7 theorems, 122 equations, 8 figures, 1 algorithm)

This paper contains 52 sections, 7 theorems, 122 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Keywords:
  2. Introduction
  3. Main contributions
  4. Summary of the main results
  5. Literature review
  6. Outline of the paper
  7. LASSO-based weighted OMP
  8. Loss function-based OMP
  9. Standard OMP.
  10. $\ell^0_w$-based Weighted OMP ($\ell^0_w$-WOMP).
  11. LASSO-type loss functions for weighted $\ell^1$ minimization
  12. Bounded noise: weighted LASSO and SR-LASSO.
  13. Sparse corruptions: weighted LAD-LASSO.
  14. Two key questions
  15. Greedy selection rules for weighted LASSO-type loss functions
  16. ...and 37 more sections

Key Result

Theorem 1

Let $\lambda \geq 0$, $S \subseteq [N]$, and $x \in \mathbb{F}^N$ satisfying Then, the loss reduction $\Delta(x,S,j)$ defined in eq:def_Delta_intro admits explicit formulas provided by eq:LASSO_WOMP_quantity, eq:SRLASSO_WOMP_quantity and eq:LADLASSO_WOMP_quantity, respectively, for the weighted LASSO, SR-LASSO and LAD-LASSO loss functions (see eq:loss_WLASSO, eq:loss_WSRLASS

Figures (8)

  • Figure 1: Relative error as a function of the tuning parameter (Experiment I, sparse random Gaussian setting). We compare the recovery accuracy of $\ell^0$- and $\ell^1$-based WOMP algorithms for different noise or corruption levels, as in \ref{['eq:nuermical_model']}.
  • Figure 2: Relative error as a function of the tuning parameter (Experiment II, function approximation). We compare the recovery accuracy of $\ell^0$- and $\ell^1$-based WOMP algorithms for different noise or corruption levels, as in \ref{['eq:nuermical_model']}.
  • Figure 3: Relative error as a function of the tuning parameter (Experiment III, sparse random Gaussian setting with oracle). Different $\ell^1$-based WOMP algorithms are tested for a fixed noise level, different choices of weights depending on the parameter $w_0$ (see \ref{['eq:oracle_weights']}), and for low (top row) and high (bottom row) values of $m$.
  • Figure 4: Relative error as a function of the iteration number (Experiment IV, sparse random Gaussian setting). The proposed $\ell^1$-based WOMP formulations are tested for different values of the tuning parameter $\lambda$. The black curve corresponds to recovery via convex optimization of the corresponding loss function.
  • Figure 5: Relative error as a function of the iteration number (Experiment V, function approximation). The proposed $\ell^1$-based WOMP formulations are tested for different values of the tuning parameter $\lambda$. The black curve corresponds to recovery via convex optimization of the corresponding loss function.
  • ...and 3 more figures

Theorems & Definitions (17)

  • Remark 1
  • Remark 2: Standard OMP
  • Theorem 1: Weighted $*$-LASSO-based greedy selection rules
  • Remark 3: $\ell^0_w$-based regularization
  • Theorem 2: LASSO-based greedy selection rule
  • Theorem 3: SR-LASSO-based greedy selection rule
  • Theorem 4: LAD-LASSO-based greedy selection rule
  • Remark 4: On terminology
  • Remark 5: Solving LAD problems
  • Remark 6: An alternative strategy
  • ...and 7 more