Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Adaptive and Parameter-Free Nesterov's Accelerated Gradient Method for Convex Optimization

Jaewook J. Suh, Shiqian Ma

TL;DR

The paper addresses adaptive, parameter-free convex optimization by introducing AdaNAG, an adaptive Nesterov-style method that is line-search-free and achieves the accelerated rate $f(x_k) - f_ = O(1/k^2)$ with a bound on gradient norms $\min_i \|\nabla f(x_i)\|^2 = O(1/k^3)$ via a Lyapunov analysis. It also derives AdaGD, a non-momentum gradient-descent-type adaptive method with non-ergodic $O(1/k)$ convergence, and develops a generalized AdaNAG family (AdaNAG-G) with practical variants (e.g., AdaNAG-G$_{12}$, AdaNAG-G$^{1/2}$) that maintain accelerated rates under locally smooth conditions. The authors provide theoretical convergence guarantees through carefully constructed Lyapunov functions and step-size rules that adapt to local smoothness $L_{k+1}$ without line searches, and validate the approach with numerical experiments in logistic regression and least-squares problems showing competitive or superior performance to recent adaptive methods like AC-FGM. The work advances adaptive acceleration by delivering non-ergodic accelerated guarantees for adaptive methods and offering practically useful generalized variants that perform well across representative applications.

Abstract

We propose AdaNAG, an adaptive accelerated gradient method based on Nesterov's accelerated gradient method. AdaNAG is line-search-free, parameter-free, and achieves the accelerated convergence rates $f(x_k) - f_\star = \mathcal{O}\left(1/k^2\right)$ and $\min_{i\in\left\{1,\dots, k\right\}} \|\nabla f(x_i)\|^2 = \mathcal{O}\left(1/k^3\right)$ for $L$-smooth convex function $f$. We provide a Lyapunov analysis for the convergence proof of AdaNAG, which additionally enables us to propose a novel adaptive gradient descent (GD) method, AdaGD. AdaGD achieves the non-ergodic convergence rate $f(x_k) - f_\star = \mathcal{O}\left(1/k\right)$, like the original GD. The analysis of AdaGD also motivated us to propose a generalized AdaNAG that includes practically useful variants of AdaNAG. Numerical results demonstrate that our methods outperform some other recent adaptive methods for representative applications.

An Adaptive and Parameter-Free Nesterov's Accelerated Gradient Method for Convex Optimization

TL;DR

The paper addresses adaptive, parameter-free convex optimization by introducing AdaNAG, an adaptive Nesterov-style method that is line-search-free and achieves the accelerated rate with a bound on gradient norms via a Lyapunov analysis. It also derives AdaGD, a non-momentum gradient-descent-type adaptive method with non-ergodic convergence, and develops a generalized AdaNAG family (AdaNAG-G) with practical variants (e.g., AdaNAG-G, AdaNAG-G) that maintain accelerated rates under locally smooth conditions. The authors provide theoretical convergence guarantees through carefully constructed Lyapunov functions and step-size rules that adapt to local smoothness without line searches, and validate the approach with numerical experiments in logistic regression and least-squares problems showing competitive or superior performance to recent adaptive methods like AC-FGM. The work advances adaptive acceleration by delivering non-ergodic accelerated guarantees for adaptive methods and offering practically useful generalized variants that perform well across representative applications.

Abstract

We propose AdaNAG, an adaptive accelerated gradient method based on Nesterov's accelerated gradient method. AdaNAG is line-search-free, parameter-free, and achieves the accelerated convergence rates and for -smooth convex function . We provide a Lyapunov analysis for the convergence proof of AdaNAG, which additionally enables us to propose a novel adaptive gradient descent (GD) method, AdaGD. AdaGD achieves the non-ergodic convergence rate , like the original GD. The analysis of AdaGD also motivated us to propose a generalized AdaNAG that includes practically useful variants of AdaNAG. Numerical results demonstrate that our methods outperform some other recent adaptive methods for representative applications.
Paper Structure (50 sections, 34 theorems, 171 equations, 5 figures, 5 tables, 4 algorithms)

This paper contains 50 sections, 34 theorems, 171 equations, 5 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Prior works
  3. Nesterov acceleration.
  4. Adaptive methods for convex function minimization.
  5. Lyapunov analysis.
  6. Contributions
  7. Organization.
  8. Preliminaries
  9. Our AdaNAG Algorithm
  10. AdaNAG
  11. Comparison with AC-FGM
  12. Convergence analysis of AdaNAG
  13. Motivation of the Lyapunov function and $L_{k+1}$
  14. Convergence analysis of AdaNAG
  15. Part 1.
  16. ...and 35 more sections

Key Result

Lemma 1

Suppose $f$ is a locally smooth convex function and $K\subset\mathbb{R}^d$ is a compact set. Let $K \subset \bar{B}_R(c)$, where $\bar{B}_R(c) = \{ x \mid \| x - c \| \leq R \}$. Let $\bar{L}_K>0$ be a smoothness parameter of $f$ on $\bar{B}_{3R}(c)$. Then the following inequality is tr Note that when $f$ is a (global) $L$-smooth convex function, the above statements hold with $\bar{L}_{K} = L$.

Figures (5)

  • Figure 1: Logistic regression problem, comparison between our algorithms
  • Figure 2: Logistic regression problem: iteration number
  • Figure 3: Logistic regression problem: CPU time
  • Figure 4: Least squares problem, synthetic datasets
  • Figure 5: Least squares problem, real-world datasets

Theorems & Definitions (61)

  • Lemma 1
  • Corollary 2
  • Theorem 1
  • Lemma 3
  • Theorem 2
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • proof
  • ...and 51 more