Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Hao Luo

TL;DR

This paper proposes new accelerated primal-dual proximal gradient splitting methods for solving bilinear saddle-point problems with provable optimal nonergodic convergence rates and presents an accelerated primal-dual hybrid gradient (APDHG) flow which combines acceleration with careful velocity correction.

Abstract

In this paper, based a novel primal-dual dynamical model with adaptive scaling parameters and Bregman divergences, we propose new accelerated primal-dual proximal gradient splitting methods for solving bilinear saddle-point problems with provable optimal nonergodic convergence rates. For the first, using the spectral analysis, we show that a naive extension of acceleration model for unconstrained optimization problems to a quadratic game is unstable. Motivated by this, we present an accelerated primal-dual hybrid gradient (APDHG) flow which combines acceleration with careful velocity correction. To work with non-Euclidean distances, we also equip our APDHG model with general Bregman divergences and prove the exponential decay of a Lyapunov function. Then, new primal-dual splitting methods are developed based on proper semi-implicit Euler schemes of the continuous model, and the theoretical convergence rates are nonergodic and optimal with respect to the matrix norms,\, Lipschitz constants and convexity parameters. Thanks to the primal and dual scaling parameters, both the algorithm designing and convergence analysis cover automatically the convex and (partially) strongly convex objectives. Moreover, the use of Bregman divergences not only unifies the standard Euclidean distances and general cases in an elegant way, but also makes our methods more flexible and adaptive to problem-dependent metrics.

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

TL;DR

This paper proposes new accelerated primal-dual proximal gradient splitting methods for solving bilinear saddle-point problems with provable optimal nonergodic convergence rates and presents an accelerated primal-dual hybrid gradient (APDHG) flow which combines acceleration with careful velocity correction.

Abstract

In this paper, based a novel primal-dual dynamical model with adaptive scaling parameters and Bregman divergences, we propose new accelerated primal-dual proximal gradient splitting methods for solving bilinear saddle-point problems with provable optimal nonergodic convergence rates. For the first, using the spectral analysis, we show that a naive extension of acceleration model for unconstrained optimization problems to a quadratic game is unstable. Motivated by this, we present an accelerated primal-dual hybrid gradient (APDHG) flow which combines acceleration with careful velocity correction. To work with non-Euclidean distances, we also equip our APDHG model with general Bregman divergences and prove the exponential decay of a Lyapunov function. Then, new primal-dual splitting methods are developed based on proper semi-implicit Euler schemes of the continuous model, and the theoretical convergence rates are nonergodic and optimal with respect to the matrix norms,\, Lipschitz constants and convexity parameters. Thanks to the primal and dual scaling parameters, both the algorithm designing and convergence analysis cover automatically the convex and (partially) strongly convex objectives. Moreover, the use of Bregman divergences not only unifies the standard Euclidean distances and general cases in an elegant way, but also makes our methods more flexible and adaptive to problem-dependent metrics.
Paper Structure (34 sections, 21 theorems, 149 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 34 sections, 21 theorems, 149 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature review
  3. Primal-dual splitting
  4. Smoothing
  5. Main contribution
  6. Organization
  7. Preliminaries
  8. Notations
  9. Bregman distance
  10. Two auxiliary functions
  11. Lyapunov functions
  12. Spectral Analysis of a Quadratic Game
  13. The unconstrained case
  14. A quadratic game
  15. NAG with velocity correction
  16. ...and 19 more sections

Key Result

Lemma 2.1

Let $\phi: {\bf dom\,}\phi\to\,{\mathbb R}$ be a prox-function, then In particular, for $\phi(x)=1/2\left\lVert {x} \right\rVert_R^2$, we have

Figures (1)

  • Figure 1: Numerical illustrations of the spectrum and the trajectories of \ref{['eq:nag-sys', 'eq:APDG-spec']}.

Theorems & Definitions (49)

  • Definition 2.1: Prox-function
  • Definition 2.2: Bregman distance
  • Lemma 2.1: luo_universal_2024
  • Definition 2.3: Relatively strong convexity
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • Lemma 2.4
  • proof
  • Lemma 2.5
  • ...and 39 more