Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Over-Relaxation Heavy-Ball Method: Achieving Global Accelerated Convergence with Broad Generalization

Jingrong Wei, Long Chen

TL;DR

The paper addresses achieving global accelerated convergence for smooth strongly convex optimization and extends this capability to composite and saddle-point problems. It introduces the Accelerated Over-Relaxation Heavy-Ball (AOR-HB) method, derived from discretizing a two-dimensional accelerated flow with an over-relaxation in the gradient term, and proves a strong Lyapunov property that yields global accelerated linear convergence with optimal iteration complexity $\mathcal{O}(\sqrt{\kappa}\,|\log\varepsilon|)$. The authors further extend AOR-HB to composite convex minimization and to a class of bilinear saddle-point problems via AOR-HB-saddle, providing convergence guarantees and practical single-loop implementations. Numerical results across smooth, composite, non-convex, and saddle-point problems illustrate robust acceleration and competitive performance relative to established first-order methods. Overall, AOR-HB offers broad generalization, conceptual clarity, and a pathway to generalize acceleration techniques beyond traditional convex settings.

Abstract

The heavy-ball momentum method accelerates gradient descent with a momentum term but lacks accelerated convergence for general smooth strongly convex problems. This work introduces the Accelerated Over-Relaxation Heavy-Ball (AOR-HB) method, the first variant with provable global and accelerated convergence for such problems. AOR-HB closes a long-standing theoretical gap, extends to composite convex optimization and min-max problems, and achieves optimal complexity bounds. It offers three key advantages: (1) broad generalization ability, (2) potential to reshape acceleration techniques, and (3) conceptual clarity and elegance compared to existing methods.

Accelerated Over-Relaxation Heavy-Ball Method: Achieving Global Accelerated Convergence with Broad Generalization

TL;DR

The paper addresses achieving global accelerated convergence for smooth strongly convex optimization and extends this capability to composite and saddle-point problems. It introduces the Accelerated Over-Relaxation Heavy-Ball (AOR-HB) method, derived from discretizing a two-dimensional accelerated flow with an over-relaxation in the gradient term, and proves a strong Lyapunov property that yields global accelerated linear convergence with optimal iteration complexity . The authors further extend AOR-HB to composite convex minimization and to a class of bilinear saddle-point problems via AOR-HB-saddle, providing convergence guarantees and practical single-loop implementations. Numerical results across smooth, composite, non-convex, and saddle-point problems illustrate robust acceleration and competitive performance relative to established first-order methods. Overall, AOR-HB offers broad generalization, conceptual clarity, and a pathway to generalize acceleration techniques beyond traditional convex settings.

Abstract

The heavy-ball momentum method accelerates gradient descent with a momentum term but lacks accelerated convergence for general smooth strongly convex problems. This work introduces the Accelerated Over-Relaxation Heavy-Ball (AOR-HB) method, the first variant with provable global and accelerated convergence for such problems. AOR-HB closes a long-standing theoretical gap, extends to composite convex optimization and min-max problems, and achieves optimal complexity bounds. It offers three key advantages: (1) broad generalization ability, (2) potential to reshape acceleration techniques, and (3) conceptual clarity and elegance compared to existing methods.
Paper Structure (43 sections, 15 theorems, 142 equations, 5 figures, 4 algorithms)

This paper contains 43 sections, 15 theorems, 142 equations, 5 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Notation.
  3. Heavy-ball methods and flow.
  4. Non-convergence and non-acceleration of the HB method.
  5. Accelerated first-order methods.
  6. Main contributions for smooth strongly convex optimization.
  7. Remark on non-strongly convex optimization.
  8. Relation to other acceleration methods.
  9. Extension to composite convex minimization.
  10. Extension to a class of saddle point problems.
  11. Extension to a class of monotone operator equations.
  12. AOR-HB Method for Convex Optimization
  13. Strong Lyapunov property.
  14. AOR-HB method for smooth strongly convex minimization problems.
  15. Preliminaries for convergence analysis.
  16. ...and 28 more sections

Key Result

Theorem 1.1

Suppose $f$ is $\mu$-strongly convex and $L$-smooth. Let $(x_k,y_k)$ be generated by scheme (AOR-HB) with initial value $(x_0, y_0)$ and step size $\alpha = \sqrt{\mu/L}$. Then there exists a constant $C_0 = C_0(x_0, y_0,\mu, L)$ so that we have the accelerated linear convergence

Figures (5)

  • Figure 1: Simulation results for the objective function (\ref{['eq:fex2']}).
  • Figure 2: Simulation results for the logistic regression problem (\ref{['eq:log reg']}).
  • Figure 3: Comparison of $\ell_1$ minimization methods.
  • Figure 4: Comparison of $\ell_1 - \ell_2$ minimization methods.
  • Figure 5: Simulation results for MSPBE (\ref{['eq:MSPBE']}).

Theorems & Definitions (29)

  • Theorem 1.1: Convergence of AOR-HB method
  • Theorem 1.2: Convergence of AOR-HB-saddle method
  • Lemma 2.1
  • proof
  • Remark 3.1
  • Lemma A.1: Section 2.1 in nesterov2018lectures
  • Lemma A.2: Bregman divergence identity chen1993convergence
  • proof
  • Lemma C.1: Strong Lyapunov property luo2022differential
  • proof
  • ...and 19 more