Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Globally Convergent Gradient Method with Momentum

Matteo Lapucci, Giampaolo Liuzzi, Stefano Lucidi, Davide Pucci, Marco Sciandrone

TL;DR

This paper tackles nonconvex unconstrained optimization by developing gradient methods with momentum whose momentum coefficients are determined by minimizing a 2×2 quadratic model in the subspace spanned by the current gradient and the previous step. The authors prove global convergence and an optimal $O(\epsilon^{-2})$ complexity bound under first-order smoothness, based on the Hessian of the 2×2 subproblem rather than full second-order information. They propose three practical schemes to compute or approximate the 2×2 Hessian $H_k$ (finite-difference, interpolation-based, and diagonal Barzilai–Borwein–style), enabling scalable large-scale implementation with a safeguarded Cholesky update. Computational experiments on large CUTEst problems show the approach is competitive with, and in some cases superior to, standard nonlinear CG and LBFGS methods, highlighting its potential as a robust baseline for momentum-based gradient optimization in nonconvex settings.

Abstract

In this work, we consider smooth unconstrained optimization problems and we deal with the class of gradient methods with momentum, i.e., descent algorithms where the search direction is defined as a linear combination of the current gradient and the preceding search direction. This family of algorithms includes nonlinear conjugate gradient methods and Polyak's heavy-ball approach, and is thus of high practical and theoretical interest in large-scale nonlinear optimization. We propose a general framework where the scalars of the linear combination defining the search direction are computed simultaneously by minimizing the approximate quadratic model in the 2 dimensional subspace. This strategy allows us to define a class of gradient methods with momentum enjoying global convergence guarantees and an optimal worst-case complexity bound in the nonconvex setting. Differently than all related works in the literature, the convergence conditions are stated in terms of the Hessian matrix of the bi-dimensional quadratic model. To the best of our knowledge, these results are novel to the literature. Moreover, extensive computational experiments show that the gradient method with momentum here presented is a solid choice to tackle some classes of nonconvex unconstrained problems.

A Globally Convergent Gradient Method with Momentum

TL;DR

This paper tackles nonconvex unconstrained optimization by developing gradient methods with momentum whose momentum coefficients are determined by minimizing a 2×2 quadratic model in the subspace spanned by the current gradient and the previous step. The authors prove global convergence and an optimal complexity bound under first-order smoothness, based on the Hessian of the 2×2 subproblem rather than full second-order information. They propose three practical schemes to compute or approximate the 2×2 Hessian (finite-difference, interpolation-based, and diagonal Barzilai–Borwein–style), enabling scalable large-scale implementation with a safeguarded Cholesky update. Computational experiments on large CUTEst problems show the approach is competitive with, and in some cases superior to, standard nonlinear CG and LBFGS methods, highlighting its potential as a robust baseline for momentum-based gradient optimization in nonconvex settings.

Abstract

In this work, we consider smooth unconstrained optimization problems and we deal with the class of gradient methods with momentum, i.e., descent algorithms where the search direction is defined as a linear combination of the current gradient and the preceding search direction. This family of algorithms includes nonlinear conjugate gradient methods and Polyak's heavy-ball approach, and is thus of high practical and theoretical interest in large-scale nonlinear optimization. We propose a general framework where the scalars of the linear combination defining the search direction are computed simultaneously by minimizing the approximate quadratic model in the 2 dimensional subspace. This strategy allows us to define a class of gradient methods with momentum enjoying global convergence guarantees and an optimal worst-case complexity bound in the nonconvex setting. Differently than all related works in the literature, the convergence conditions are stated in terms of the Hessian matrix of the bi-dimensional quadratic model. To the best of our knowledge, these results are novel to the literature. Moreover, extensive computational experiments show that the gradient method with momentum here presented is a solid choice to tackle some classes of nonconvex unconstrained problems.
Paper Structure (14 sections, 7 theorems, 85 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 14 sections, 7 theorems, 85 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. The main idea
  3. Related works and open problems
  4. Existence of a solution for the two dimensional problem
  5. Properties of the search directions
  6. Conditions on ${B_k}$
  7. Conditions on $H_k$
  8. Algorithmic model
  9. Computation of ${H_k}$
  10. Approximating Hessian-vector products by finite differences of gradients
  11. Hessian estimation in subspace by interpolation
  12. Hessian approximation by a diagonal matrix
  13. Computational experiments
  14. Conclusions

Key Result

Proposition 1

Consider problem eq:constrained_dir_prob2b. If $H_k$ is a symmetric positive definite matrix, then problem eq:constrained_dir_prob2b admits an optimal solution.

Figures (1)

  • Figure 1: Performance profiles for the comparison between GMM, cg_descent and the Scipy's implementations of the conjugate gradient method and L-BFGS, on the 77 problems considered.

Theorems & Definitions (18)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Definition 1
  • Proposition 3
  • proof
  • Proposition 4
  • proof
  • Remark 1
  • ...and 8 more