Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparison of Proximal First-Order Primal and Primal-Dual algorithms via Performance Estimation

Nizar Bousselmi, Nelly Pustelnik, Julien M. Hendrickx, François Glineur

TL;DR

An approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions is proposed.

Abstract

Selecting the fastest algorithm for a specific signal/image processing task is a challenging question. We propose an approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions. We first propose a computer-assisted analysis and comparison of several first-order primal optimization methods, namely, the gradient method, the forward-backward, Peaceman-Rachford, and Douglas-Rachford splittings. We tighten the existing convergence results of these algorithms and extend them to new classes of functions. Our analysis is then extended and evaluated in the context of the primal-dual Chambolle-Pock and Condat-Vũ methods.

Comparison of Proximal First-Order Primal and Primal-Dual algorithms via Performance Estimation

TL;DR

An approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions is proposed.

Abstract

Selecting the fastest algorithm for a specific signal/image processing task is a challenging question. We propose an approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions. We first propose a computer-assisted analysis and comparison of several first-order primal optimization methods, namely, the gradient method, the forward-backward, Peaceman-Rachford, and Douglas-Rachford splittings. We tighten the existing convergence results of these algorithms and extend them to new classes of functions. Our analysis is then extended and evaluated in the context of the primal-dual Chambolle-Pock and Condat-Vũ methods.
Paper Structure (11 sections, 3 theorems, 11 equations, 5 figures)

This paper contains 11 sections, 3 theorems, 11 equations, 5 figures.

Table of Contents

  1. Introduction
  2. PEP analysis for the composite problem $f+g$
  3. First-order proximal methods to solve \ref{['eq:problem1']}
  4. PEP to validate the tightness of theoretical analysis
  5. PEP to evaluate the impact of changing the regularization parameter in the objective function
  6. Focus on Douglas-Rachford
  7. Generalization to strongly convex $f$ and $g$
  8. Linear rate on quadratic functions
  9. Case $f+h\circ M$
  10. Primal-dual methods
  11. Conclusion

Key Result

Proposition 1

Each upper bound eq:r_GM, eq:r_FBS1, eq:r_FBS2, and eq:r_PRS is attained by a quadratic function, and hence describes the exact (unimprovable) worst-case contraction factors of (GM), (FBS1), (FBS2), and (PRS).

Figures (5)

  • Figure 1: Comparison of the contraction factors from briceno2023theoretical (solid lines) and PEP (dots) of GM (blue), FBS1 (red), FBS2 (green), PRS (magenta), DRS (purple), and the optimal rate predicted by briceno2023theoretical (black circle) and PEP (black square).
  • Figure 2: Worst-case contraction factor of DRS for $\alpha=1$, $\beta = 5$, $\rho = 0.1$, $\mu=0$ from briceno2023theoretical (purple solid line) and PEP (purple dots). We identified two regimes (blue and red solid lines).
  • Figure 3: Contraction factors obtained by PEP of GM (blue), FBS1 (red), FBS2 (green), PRS (magenta), DRS (purple) in the doubly strongly convex case $\alpha = 1$, $\beta = 5$, $\rho=\mu=0.1$.
  • Figure 4: Contraction factor of DRS provided by PEP on \ref{['eq:problem1']} (purple dots) and \ref{['eq:problem2']} (purple squares) for $\alpha=1$, $\beta = \gamma = 5$, $\rho = 0.1$, $\mu = 0$, $\delta = 0.1$ and $L = 1$.
  • Figure 5: Contraction factors of CPM (blue) and CVM (red) obtained by PEP for $\alpha = 1$, $\beta=5$, $\rho=0.1$, $\delta=0$ (squares) and $\delta = 0.1$ (dots) and $||M||\leq 1$.

Theorems & Definitions (7)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • Conjecture 1