Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic behavior for a gradient algorithm with energy and momentum

Hailiang Liu, Xuping Tian

TL;DR

The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories and establishes a linear convergence rate for objective functions that adhere to the Polyak-{\L}ojasiewicz condition.

Abstract

This paper investigates a novel gradient algorithm, AGEM, using both energy and momentum, for addressing general non-convex optimization problems. The solution properties of the AGEM algorithm, including aspects such as uniformly boundedness and convergence to critical points, are examined. The dynamic behavior is studied through a comprehensive analysis of a high-resolution ODE system. This ODE system, being nonlinear, is derived by taking the limit of the discrete scheme while preserving the momentum effect through a rescaling of the momentum parameter. The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories. Furthermore, we establish a linear convergence rate for objective functions that adhere to the Polyak-Łojasiewicz condition.

Dynamic behavior for a gradient algorithm with energy and momentum

TL;DR

The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories and establishes a linear convergence rate for objective functions that adhere to the Polyak-{\L}ojasiewicz condition.

Abstract

This paper investigates a novel gradient algorithm, AGEM, using both energy and momentum, for addressing general non-convex optimization problems. The solution properties of the AGEM algorithm, including aspects such as uniformly boundedness and convergence to critical points, are examined. The dynamic behavior is studied through a comprehensive analysis of a high-resolution ODE system. This ODE system, being nonlinear, is derived by taking the limit of the discrete scheme while preserving the momentum effect through a rescaling of the momentum parameter. The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories. Furthermore, we establish a linear convergence rate for objective functions that adhere to the Polyak-Łojasiewicz condition.
Paper Structure (15 sections, 9 theorems, 158 equations, 2 figures)

This paper contains 15 sections, 9 theorems, 158 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related works.
  3. Energy-adaptive gradient algorithms
  4. Energy-adaptive gradient algorithms.
  5. PL condition for non-convex objectives.
  6. Solution properties of the AGEM algorithm
  7. Discrete schemes vs continuous-time-limits
  8. High resolution ODEs.
  9. Time discretization.
  10. Dynamic solution behavior
  11. Global Existence and Uniqueness.
  12. Asymptotic behavior of solutions.
  13. Convergence rates
  14. Convergence of trajectories
  15. Discussion

Key Result

theorem 1

AEGD (aegd), (aegdm), (gdem), and (sgdem) all exhibit unconditionally energy stability, as expressed by the relation: This guarantees the strict decrease of $r_k$ with convergence to $r^*\geq0$ as $k\to\infty$, accompanied by the limit $\lim_{k\to\infty}|\theta_{k+1}-\theta_k|^2=0$. Furthermore, the following inequalities hold:

Figures (2)

  • Figure 1: The number of iterations required for the three energy-adaptive methods AEGDM, SGEM, AGEM, and GDM, to converge to the global minima of the two-dimensional Rosenbrock function, across different values of $\beta$ and $\eta$. In the case of GDM, the white areas on the graph indicate divergences.
  • Figure 2: A comparative analysis showcasing the performance of various methods on the Rosenbrock function.

Theorems & Definitions (26)

  • theorem 1: Unconditional energy stability
  • proof
  • definition 1
  • theorem 2
  • proof
  • remark 1
  • theorem 3
  • theorem 4
  • proof
  • remark 2
  • ...and 16 more