Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated tight Lyapunov analysis for first-order methods

Manu Upadhyaya, Sebastian Banert, Adrien B. Taylor, Pontus Giselsson

TL;DR

This work automates Lyapunov-based convergence analysis for a broad class of fixed-parameter first-order methods solving convex problems by casting the search for a quadratic Lyapunov inequality as a small semidefinite program. It unifies an operator-splitting algorithm representation with interpolation conditions and derives a necessary-and-sufficient SDP feasibility test (Theorem) that certifies Lyapunov functions and residuals over the entire function class. The approach yields both linear and ergodic convergence guarantees and extends duality-gap convergence regions for challenging methods such as Chambolle–Pock when the linear operator is the identity, beyond classical parameter choices. The methodology, validated on Douglas–Rachford, heavy-ball gradient variants, Davis–Yin, and Chambolle–Pock, provides sharper rates and wider operational parameter ranges with practical implications for designing and tuning first-order optimization algorithms.

Abstract

We present a methodology for establishing the existence of quadratic Lyapunov inequalities for a wide range of first-order methods used to solve convex optimization problems. In particular, we consider i) classes of optimization problems of finite-sum form with (possibly strongly) convex and possibly smooth functional components, ii) first-order methods that can be written as a linear system in state-space form in feedback interconnection with the subdifferentials of the functional components of the objective function, and iii) quadratic Lyapunov inequalities that can be used to draw convergence conclusions. We present a necessary and sufficient condition for the existence of a quadratic Lyapunov inequality within a predefined class of Lyapunov inequalities, which amounts to solving a small-sized semidefinite program. We showcase our methodology on several first-order methods that fit the framework. Most notably, our methodology allows us to significantly extend the region of parameter choices that allow for duality-gap convergence in the Chambolle-Pock method when the linear operator is the identity mapping.

Automated tight Lyapunov analysis for first-order methods

TL;DR

This work automates Lyapunov-based convergence analysis for a broad class of fixed-parameter first-order methods solving convex problems by casting the search for a quadratic Lyapunov inequality as a small semidefinite program. It unifies an operator-splitting algorithm representation with interpolation conditions and derives a necessary-and-sufficient SDP feasibility test (Theorem) that certifies Lyapunov functions and residuals over the entire function class. The approach yields both linear and ergodic convergence guarantees and extends duality-gap convergence regions for challenging methods such as Chambolle–Pock when the linear operator is the identity, beyond classical parameter choices. The methodology, validated on Douglas–Rachford, heavy-ball gradient variants, Davis–Yin, and Chambolle–Pock, provides sharper rates and wider operational parameter ranges with practical implications for designing and tuning first-order optimization algorithms.

Abstract

We present a methodology for establishing the existence of quadratic Lyapunov inequalities for a wide range of first-order methods used to solve convex optimization problems. In particular, we consider i) classes of optimization problems of finite-sum form with (possibly strongly) convex and possibly smooth functional components, ii) first-order methods that can be written as a linear system in state-space form in feedback interconnection with the subdifferentials of the functional components of the objective function, and iii) quadratic Lyapunov inequalities that can be used to draw convergence conclusions. We present a necessary and sufficient condition for the existence of a quadratic Lyapunov inequality within a predefined class of Lyapunov inequalities, which amounts to solving a small-sized semidefinite program. We showcase our methodology on several first-order methods that fit the framework. Most notably, our methodology allows us to significantly extend the region of parameter choices that allow for duality-gap convergence in the Chambolle-Pock method when the linear operator is the identity mapping.
Paper Structure (31 sections, 6 theorems, 105 equations, 4 figures)

This paper contains 31 sections, 6 theorems, 105 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem class and algorithm representation
  4. Problem class
  5. Algorithm representation
  6. Solutions and fixed points
  7. Well-posedness
  8. Examples
  9. Douglas--Rachford method
  10. Gradient method with heavy-ball momentum
  11. Proximal gradient method with heavy-ball momentum terms
  12. Davis--Yin three-operator splitting method
  13. Chambolle--Pock method
  14. Interpolation
  15. Lyapunov inequalities
  16. ...and 16 more sections

Key Result

Proposition 1

The following are equivalent:

Figures (4)

  • Figure 1: B\ref{['item:numeric_linear']} applied to the Douglas–Rachford method (see Section \ref{['sec:douglas_rachford']}) when $f_1\in\mathcal{F}_{1,2}$, $f_2\in\mathcal{F}_{0,\infty}$ and $\lambda = 1$, and the tight convergence rate given in Giselsson2017Linear.
  • Figure 2: Convergence analysis of the (proximal) gradient method with heavy-ball momentum (see Section \ref{['sec:gradient_method_heavy-ball_momentum']} and Section \ref{['sec:proximal_gradient_method_heavy-ball_momentum']}).
  • Figure 3: B\ref{['item:numeric_linear']} applied to the three-operator splitting method by Davis and Yin (see Section \ref{['sec:davis_yin']}) when $f_1\in\mathcal{F}_{0,\beta_1}$, $f_2\in\mathcal{F}_{1,2}$, $f_3\in\mathcal{F}_{0,\infty}$, $\gamma = 1/2$ and $\lambda = 1$, the linear convergence rate given in Davis_Yin_3op_arxiv, and the linear convergence rate given in pmlr-v80-pedregosa18a.
  • Figure 4: Convergence analysis of the Chambolle--Pock method (see Section \ref{['sec:chambolle_pock']}).

Theorems & Definitions (17)

  • Definition 1: Fixed-point encoding property
  • Proposition 1
  • proof
  • Remark 1
  • Remark 2
  • Proposition 2
  • proof
  • Theorem 1
  • Corollary 1
  • Definition 2: Algorithm consistency
  • ...and 7 more