Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Gradient Methods Through Variable and Operator Splitting

Long Chen, Luo Hao, Jingrong Wei

TL;DR

The paper develops a unified Variable and Operator Splitting (VOS) framework to accelerate first-order methods across convex, composite, and saddle-point problems by decoupling optimization via a secondary variable and operator-splitting. Central to the approach are strong Lyapunov functions that yield convergence guarantees for continuous flows and their discretizations, notably Accelerated Over-Relaxation (AOR) and Extrapolation by Predictor-Corrector (EPC), plus dynamic/scaled and perturbed variants. The authors show linear, exponential, and sublinear rates in various regimes, with concrete schemes such as AOR-VOS, EPC-VOS, and Accelerated Gradient-Skew-Symmetric Splitting (AGSS), achieving optimal first-order complexity in several settings. This framework unifies and extends acceleration techniques to a broad class of problems, including strongly convex, composite convex, and saddle-point systems with bilinear coupling, offering practical, provably-fast methods with clear discretization strategies.

Abstract

This paper introduces a unified framework for accelerated gradient methods through the variable and operator splitting (VOS). The operator splitting decouples the optimization process into simpler subproblems, and more importantly, the variable splitting leads to acceleration. The key contributions include the development of strong Lyapunov functions to analyze stability and convergence rates, as well as advanced discretization techniques like Accelerated Over-Relaxation (AOR) and extrapolation by the predictor-corrector methods (EPC). For convex case, we introduce a dynamic updating parameter and a perturbed VOS flow. The framework effectively handles a wide range of optimization problems, including convex optimization, composite convex optimization, and saddle point systems with bilinear coupling.

Accelerated Gradient Methods Through Variable and Operator Splitting

TL;DR

The paper develops a unified Variable and Operator Splitting (VOS) framework to accelerate first-order methods across convex, composite, and saddle-point problems by decoupling optimization via a secondary variable and operator-splitting. Central to the approach are strong Lyapunov functions that yield convergence guarantees for continuous flows and their discretizations, notably Accelerated Over-Relaxation (AOR) and Extrapolation by Predictor-Corrector (EPC), plus dynamic/scaled and perturbed variants. The authors show linear, exponential, and sublinear rates in various regimes, with concrete schemes such as AOR-VOS, EPC-VOS, and Accelerated Gradient-Skew-Symmetric Splitting (AGSS), achieving optimal first-order complexity in several settings. This framework unifies and extends acceleration techniques to a broad class of problems, including strongly convex, composite convex, and saddle-point systems with bilinear coupling, offering practical, provably-fast methods with clear discretization strategies.

Abstract

This paper introduces a unified framework for accelerated gradient methods through the variable and operator splitting (VOS). The operator splitting decouples the optimization process into simpler subproblems, and more importantly, the variable splitting leads to acceleration. The key contributions include the development of strong Lyapunov functions to analyze stability and convergence rates, as well as advanced discretization techniques like Accelerated Over-Relaxation (AOR) and extrapolation by the predictor-corrector methods (EPC). For convex case, we introduce a dynamic updating parameter and a perturbed VOS flow. The framework effectively handles a wide range of optimization problems, including convex optimization, composite convex optimization, and saddle point systems with bilinear coupling.
Paper Structure (39 sections, 31 theorems, 273 equations, 1 figure, 1 algorithm)

This paper contains 39 sections, 31 theorems, 273 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Bounds on Convex Functions
  3. Convex functions
  4. Bregman divergence and various bounds
  5. Conjugate function
  6. Bounds involving a global minimum
  7. Strong Lyapunov Functions
  8. Strong Lyapunov condition and decay property
  9. Linear convergence of the implicit Euler method
  10. Gradient Flow and Euler Methods
  11. Gradient Flow
  12. Strongly convex case
  13. Convex case
  14. Proximal Point Algorithm
  15. Gradient Descent Methods
  16. ...and 24 more sections

Key Result

Theorem 2.1

If $f$ is convex and coercive, then $\min_{x\in V}f(x)$ admits at least one solution $x^{\star}\in V$, which is unique if we assume further $f$ is strictly convex.

Figures (1)

  • Figure 6.1: Extrapolation by the predictor-corrector methods. Triangle $\Delta(x_k, \tilde{x}_{k+1}, x_{k+1})$ and triangle $\Delta(x_k, y_k, y_{k+1})$ are similar.

Theorems & Definitions (38)

  • Theorem 2.1
  • Lemma 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Corollary 2.1
  • Corollary 2.2
  • Theorem 3.1
  • Theorem 3.2
  • Proposition 4.1
  • Lemma 4.1
  • ...and 28 more