Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast sparse optimization via adaptive shrinkage

Vito Cerone, Sophie M. Fosson, Diego Regruto

TL;DR

The paper tackles fast sparse optimization in linear models by introducing Log-Lasso, a non-convex regularized variant solved via a proximal gradient framework. It derives AD-ISTA, an adaptive shrinkage ISTA variant, showing equivalence to ISTA with a data-driven shrinkage parameter and proving monotonic improvement of the objective. An accelerated version, AD-FISTA, is developed using FISTA techniques to achieve faster convergence, and comparisons to RW-ISTA and other methods illustrate superior speed in practice. Numerical experiments demonstrate that AD-ISTA and especially AD-FISTA converge in far fewer iterations than classical methods while reaching the same high-quality sparse solutions, indicating strong practical impact for fast sparse recovery and time-varying system identification.

Abstract

The need for fast sparse optimization is emerging, e.g., to deal with large-dimensional data-driven problems and to track time-varying systems. In the framework of linear sparse optimization, the iterative shrinkage-thresholding algorithm is a valuable method to solve Lasso, which is particularly appreciated for its ease of implementation. Nevertheless, it converges slowly. In this paper, we develop a proximal method, based on logarithmic regularization, which turns out to be an iterative shrinkage-thresholding algorithm with adaptive shrinkage hyperparameter. This adaptivity substantially enhances the trajectory of the algorithm, in a way that yields faster convergence, while keeping the simplicity of the original method. Our contribution is twofold: on the one hand, we derive and analyze the proposed algorithm; on the other hand, we validate its fast convergence via numerical experiments and we discuss the performance with respect to state-of-the-art algorithms.

Fast sparse optimization via adaptive shrinkage

TL;DR

The paper tackles fast sparse optimization in linear models by introducing Log-Lasso, a non-convex regularized variant solved via a proximal gradient framework. It derives AD-ISTA, an adaptive shrinkage ISTA variant, showing equivalence to ISTA with a data-driven shrinkage parameter and proving monotonic improvement of the objective. An accelerated version, AD-FISTA, is developed using FISTA techniques to achieve faster convergence, and comparisons to RW-ISTA and other methods illustrate superior speed in practice. Numerical experiments demonstrate that AD-ISTA and especially AD-FISTA converge in far fewer iterations than classical methods while reaching the same high-quality sparse solutions, indicating strong practical impact for fast sparse recovery and time-varying system identification.

Abstract

The need for fast sparse optimization is emerging, e.g., to deal with large-dimensional data-driven problems and to track time-varying systems. In the framework of linear sparse optimization, the iterative shrinkage-thresholding algorithm is a valuable method to solve Lasso, which is particularly appreciated for its ease of implementation. Nevertheless, it converges slowly. In this paper, we develop a proximal method, based on logarithmic regularization, which turns out to be an iterative shrinkage-thresholding algorithm with adaptive shrinkage hyperparameter. This adaptivity substantially enhances the trajectory of the algorithm, in a way that yields faster convergence, while keeping the simplicity of the original method. Our contribution is twofold: on the one hand, we derive and analyze the proposed algorithm; on the other hand, we validate its fast convergence via numerical experiments and we discuss the performance with respect to state-of-the-art algorithms.
Paper Structure (9 sections, 2 theorems, 21 equations, 2 figures, 1 table)

This paper contains 9 sections, 2 theorems, 21 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Proposed approach
  4. Why AD-ISTA is faster than ISTA?
  5. Related algorithms
  6. AD-FISTA: a fast version of AD-ISTA
  7. Comparison to $\ell_1$-reweighting ISTA
  8. Numerical results
  9. Conclusions

Key Result

Lemma 1

Given the sequence $x_t$ generated by PGM applied to p2, $\mathcal{F}(x_t)$ converges. Moreover, PGM applied to p2 defines an asymptotically regular map, that is, $\lim\limits_{t\to\infty}\|x_{t+1}-x_t\|_2=0$.

Figures (2)

  • Figure 1: Residual norm $\|Ax_t-y\|_2$ with respect to $\|x_t\|_1$ and $\|x_t\|_0$, respectively. The curve are parametrized with time. We label the iterations 1,10,20,50,200,500 to ease the comparison of the algorithms. "True" refers to the value of $\widetilde{x}$, which is estimated by Lasso with a small bias on the non-zero components.
  • Figure 2: The time evolution of $\|Ax_t-y\|_2$, $\|x_t\|_1$ and $\|x_t\|_0$.

Theorems & Definitions (3)

  • Lemma 1
  • Lemma 2
  • Remark 1