Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Field Guide to Forward-Backward Splitting with a FASTA Implementation

Tom Goldstein, Christoph Studer, Richard Baraniuk

TL;DR

<3-5 sentence high-level summary> This article surveys forward-backward splitting (FBS) with a practical emphasis, detailing convergence theory, adaptive strategies, and a broad spectrum of convex (and select non-convex) applications solvable via FBS. It introduces FASTA, a unified solver that enables fair comparisons across FBS variants and supports automatic parameter tuning, backtracking, continuation, and stopping criteria. Through extensive numerical experiments on problems ranging from Lasso and MMV to total variation denoising, SVM, PhaseLift, and matrix factorization, the paper demonstrates that adaptive stepsizes and acceleration can substantially improve convergence, while highlighting the importance of restart and stopping rules for stability. The work argues that FBS, when implemented with care, is a versatile and efficient alternative to more complex splitting methods for many modern optimization tasks.

Abstract

Non-differentiable and constrained optimization play a key role in machine learning, signal and image processing, communications, and beyond. For high-dimensional minimization problems involving large datasets or many unknowns, the forward-backward splitting method provides a simple, practical solver. Despite its apparently simplicity, the performance of the forward-backward splitting is highly sensitive to implementation details. This article is an introductory review of forward-backward splitting with a special emphasis on practical implementation concerns. Issues like stepsize selection, acceleration, stopping conditions, and initialization are considered. Numerical experiments are used to compare the effectiveness of different approaches. Many variations of forward-backward splitting are implemented in the solver FASTA (short for Fast Adaptive Shrinkage/Thresholding Algorithm). FASTA provides a simple interface for applying forward-backward splitting to a broad range of problems.

A Field Guide to Forward-Backward Splitting with a FASTA Implementation

TL;DR

<3-5 sentence high-level summary> This article surveys forward-backward splitting (FBS) with a practical emphasis, detailing convergence theory, adaptive strategies, and a broad spectrum of convex (and select non-convex) applications solvable via FBS. It introduces FASTA, a unified solver that enables fair comparisons across FBS variants and supports automatic parameter tuning, backtracking, continuation, and stopping criteria. Through extensive numerical experiments on problems ranging from Lasso and MMV to total variation denoising, SVM, PhaseLift, and matrix factorization, the paper demonstrates that adaptive stepsizes and acceleration can substantially improve convergence, while highlighting the importance of restart and stopping rules for stability. The work argues that FBS, when implemented with care, is a versatile and efficient alternative to more complex splitting methods for many modern optimization tasks.

Abstract

Non-differentiable and constrained optimization play a key role in machine learning, signal and image processing, communications, and beyond. For high-dimensional minimization problems involving large datasets or many unknowns, the forward-backward splitting method provides a simple, practical solver. Despite its apparently simplicity, the performance of the forward-backward splitting is highly sensitive to implementation details. This article is an introductory review of forward-backward splitting with a special emphasis on practical implementation concerns. Issues like stepsize selection, acceleration, stopping conditions, and initialization are considered. Numerical experiments are used to compare the effectiveness of different approaches. Many variations of forward-backward splitting are implemented in the solver FASTA (short for Fast Adaptive Shrinkage/Thresholding Algorithm). FASTA provides a simple interface for applying forward-backward splitting to a broad range of problems.

Paper Structure

This paper contains 44 sections, 1 theorem, 76 equations, 1 figure, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Outline
  3. A Word on Notation
  4. Forward-Backward Splitting
  5. Convergence of FBS
  6. Applications of Forward-Backward Splitting
  7. Simple Applications
  8. Convex Constraints and the Projected Gradient Method
  9. Lasso Regression
  10. $\boldsymbol \ell_1$-Norm Penalized (Logistic) Regression
  11. Multiple Measurement Vector (MMV)
  12. Democratic Representations
  13. Low-Rank (1-bit) Matrix Completion
  14. More Complex Applications
  15. Total-Variation Denoising
  16. ...and 29 more sections

Key Result

Theorem 1

Suppose that FBS is applied to general with convex $g$ and differentiable $f$. Suppose further that $h=f+g$ is proper, lower semi-continuous, and has bounded level sets. If $\{\tau^k\}$ is bounded below by a positive constant and then $\lim_{k\to\infty} h(x^k) = h^\star$, where $h^\star$ denotes the minimum value of $h$.

Figures (1)

  • Figure 1: Sample converge curves for FBS, FBS+acceleration (FISTA), and FBS+adaptivity (SpaRSA) for four diverse test problems (see Section \ref{['sec:apps']} and \ref{['sec:num']} for details). The vertical axis shows the optimality gap.

Theorems & Definitions (2)

  • Theorem 1
  • proof