Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HNAG++: A Super-Fast Accelerated Gradient Method for Strongly Convex Optimization

Long Chen, Zeyi Xu

TL;DR

The paper tackles accelerating first-order methods for strongly convex optimization with large condition numbers by deriving HNAG, HNAG+, and HNAG++ from Hessian-driven Nesterov flows. Central to the approach are double shifts in primal and dual spaces and a strong Lyapunov framework that yields provable linear convergence rates, culminating in a global rate of $1-2/\sqrt{\kappa}$ for HNAG+ and $1-2\sqrt{2}/\sqrt{\kappa}$ asymptotically for HNAG++. The methods are supported by nonasymptotic and asymptotic analyses, including a refined shifting strategy and consideration of Bregman divergence asymmetry. Numerical experiments across linear and nonlinear problems show HNAG++ consistently outperforms existing accelerated first-order methods, suggesting it is the fastest known globally convergent method in this class. The work blends ODE dynamics, Lyapunov stability, and dual-space shifting to push the limits of first-order acceleration.

Abstract

We introduce and analyze two methods, HNAG+ and HNAG++, for minimizing strongly convex functions with large condition number kappa. For HNAG+, we prove a global linear convergence rate of 1 - 2/sqrt(kappa), achieving the information-theoretic optimal rate. For HNAG++, we establish a global asymptotic linear rate of 1 - 2*sqrt(2/kappa) for functions with Hölder continuous Hessians, representing the fastest known rate among globally convergent first-order methods. Extensive numerical experiments on linear and nonlinear problems show that HNAG++ consistently outperforms existing accelerated gradient methods.

HNAG++: A Super-Fast Accelerated Gradient Method for Strongly Convex Optimization

TL;DR

The paper tackles accelerating first-order methods for strongly convex optimization with large condition numbers by deriving HNAG, HNAG+, and HNAG++ from Hessian-driven Nesterov flows. Central to the approach are double shifts in primal and dual spaces and a strong Lyapunov framework that yields provable linear convergence rates, culminating in a global rate of for HNAG+ and asymptotically for HNAG++. The methods are supported by nonasymptotic and asymptotic analyses, including a refined shifting strategy and consideration of Bregman divergence asymmetry. Numerical experiments across linear and nonlinear problems show HNAG++ consistently outperforms existing accelerated first-order methods, suggesting it is the fastest known globally convergent method in this class. The work blends ODE dynamics, Lyapunov stability, and dual-space shifting to push the limits of first-order acceleration.

Abstract

We introduce and analyze two methods, HNAG+ and HNAG++, for minimizing strongly convex functions with large condition number kappa. For HNAG+, we prove a global linear convergence rate of 1 - 2/sqrt(kappa), achieving the information-theoretic optimal rate. For HNAG++, we establish a global asymptotic linear rate of 1 - 2*sqrt(2/kappa) for functions with Hölder continuous Hessians, representing the fastest known rate among globally convergent first-order methods. Extensive numerical experiments on linear and nonlinear problems show that HNAG++ consistently outperforms existing accelerated gradient methods.
Paper Structure (27 sections, 17 theorems, 147 equations, 3 figures, 2 tables, 2 algorithms)

This paper contains 27 sections, 17 theorems, 147 equations, 3 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature review
  3. Approaches
  4. Contribution
  5. Preliminaries
  6. Main idea
  7. Outline
  8. HNAG: Method and Convergence
  9. Flow and Discretization
  10. Algorithm
  11. Strong Lyapunov Property
  12. Convergence analysis
  13. Convergence analysis using stability analsysis
  14. HNAG+: Method and Convergence
  15. Flow and Discretization
  16. ...and 12 more sections

Key Result

Lemma 1

Let $\mathcal{G}(\boldsymbol z)$ be the vector field of the HNAG flow eq:flow. The Lyapunov function $E(\boldsymbol z)$ defined in eq:lyapunov_strong_HNAG satisfies the following strong Lyapunov property

Figures (3)

  • Figure 1: Error curves for $h=1/40$ and $\kappa = 3150$.
  • Figure 2: Error curves for $\min f$\ref{['eq:fex2']} with $\mu=1, L=10^4, d=100, p=5$, and $\varepsilon = 10^{-6}$.
  • Figure 3: Error curves for $\min f$\ref{['eq:log reg']} with $\lambda =0.1, d = 1000,$ and $n = 50$.

Theorems & Definitions (35)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 3: $(1-\sqrt{2/\kappa})$-linear convergence
  • proof
  • Remark 1
  • Lemma 4
  • Theorem 5: $(1-\sqrt{2/\kappa})$-linear convergence of $\|x - x^{\star}\|^2$
  • proof
  • ...and 25 more