Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HOME-3: High-Order Momentum Estimator with Third-Power Gradient for Convex and Smooth Nonconvex Optimization

Wei Zhang, Arif Hassan Zidan, Afrar Jahin, Yu Bao, Tianming Liu

TL;DR

HOME-3 introduces a high-order momentum optimization framework that leverages the cube of the first-order gradient, $g_t^3$, to form a third-order momentum term within an Adam-like update. The method combines adaptive learning rates, bias corrections, and coordinate randomization to enhance convergence for convex and smooth nonconvex problems, with empirical extensions to nonsmooth nonconvex tasks. Theoretical results establish $O(1/T^{5/6})$ convergence bounds in both convex and smooth nonconvex settings, while coordinate randomization helps preserve these rates in nonsmooth scenarios. Across convex, smooth nonconvex, and nonsmooth nonconvex experiments, HOME-3 consistently outperforms standard momentum-based optimizers, validating the value of high-order gradient information in momentum updates.

Abstract

Momentum-based gradients are essential for optimizing advanced machine learning models, as they not only accelerate convergence but also advance optimizers to escape stationary points. While most state-of-the-art momentum techniques utilize lower-order gradients, such as the squared first-order gradient, there has been limited exploration of higher-order gradients, particularly those raised to powers greater than two. In this work, we introduce the concept of high-order momentum, where momentum is constructed using higher-power gradients, with a focus on the third-power of the first-order gradient as a representative case. Our research offers both theoretical and empirical support for this approach. Theoretically, we demonstrate that incorporating third-power gradients can improve the convergence bounds of gradient-based optimizers for both convex and smooth nonconvex problems. Empirically, we validate these findings through extensive experiments across convex, smooth nonconvex, and nonsmooth nonconvex optimization tasks. Across all cases, high-order momentum consistently outperforms conventional low-order momentum methods, showcasing superior performance in various optimization problems.

HOME-3: High-Order Momentum Estimator with Third-Power Gradient for Convex and Smooth Nonconvex Optimization

TL;DR

HOME-3 introduces a high-order momentum optimization framework that leverages the cube of the first-order gradient, , to form a third-order momentum term within an Adam-like update. The method combines adaptive learning rates, bias corrections, and coordinate randomization to enhance convergence for convex and smooth nonconvex problems, with empirical extensions to nonsmooth nonconvex tasks. Theoretical results establish convergence bounds in both convex and smooth nonconvex settings, while coordinate randomization helps preserve these rates in nonsmooth scenarios. Across convex, smooth nonconvex, and nonsmooth nonconvex experiments, HOME-3 consistently outperforms standard momentum-based optimizers, validating the value of high-order gradient information in momentum updates.

Abstract

Momentum-based gradients are essential for optimizing advanced machine learning models, as they not only accelerate convergence but also advance optimizers to escape stationary points. While most state-of-the-art momentum techniques utilize lower-order gradients, such as the squared first-order gradient, there has been limited exploration of higher-order gradients, particularly those raised to powers greater than two. In this work, we introduce the concept of high-order momentum, where momentum is constructed using higher-power gradients, with a focus on the third-power of the first-order gradient as a representative case. Our research offers both theoretical and empirical support for this approach. Theoretically, we demonstrate that incorporating third-power gradients can improve the convergence bounds of gradient-based optimizers for both convex and smooth nonconvex problems. Empirically, we validate these findings through extensive experiments across convex, smooth nonconvex, and nonsmooth nonconvex optimization tasks. Across all cases, high-order momentum consistently outperforms conventional low-order momentum methods, showcasing superior performance in various optimization problems.
Paper Structure (17 sections, 4 theorems, 50 equations, 6 figures, 3 tables)

This paper contains 17 sections, 4 theorems, 50 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries: Definitions and Assumptions
  3. Problem Setup
  4. Definitions
  5. Assumptions
  6. Method: High-Order Momentum Estimator (HOME)
  7. Theoretical Analyses
  8. Convex Case
  9. Smooth Nonconvex Case
  10. Nonsmooth Nonconvex Case
  11. Numerical Experiments
  12. Experiment on Convex Problem: Dictionary Learning
  13. Experiment on Smooth Nonconvex Problem: Deep Nonlinear Matrix Factorizations
  14. Experiment on Nonsmooth Nonconvex Problem: Noisy Deep Matrix Factorization
  15. Statistical Analyses
  16. ...and 2 more sections

Key Result

Theorem 4.1

Let $f$ satisfy Assumptionassumption1, suppose $T$ as the maximum iteration, according to Definitionsdef3, def5, and def6, then $\frac{\left \| \Sigma_{t=1}^T (f(x_t)-f(x_T)) \right \|}{T} = O(1 \slash T^{5 \slash 6})$.

Figures (6)

  • Figure 1: Averaged reconstruction loss comparison of proposed HOME-3 and other three peer optimizers within one hundred iterations
  • Figure 2: Averaged reconstruction loss comparison of proposed HOME-3 and other three peer optimizers with in one hundred iterations at first and second layers of DNMF
  • Figure 3: The averaged training loss comparison of proposed HOME-3 and other three peer optimizers within one hundred iterations of all subjects at first and second layers of noisy DNMF, respectively.
  • Figure 4: Consistency and robustness comparisons of the proposed HOME-3 and three peer algorithms are presented.
  • Figure 5: An illustration of reconstruction loss comparisons of HOME-3 and other peer optimizers on solving logistic regression problem.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • Definition 2.6
  • Definition 2.7
  • Theorem 4.1
  • Theorem 4.2
  • Lemma 4.3
  • ...and 1 more