Adaptive Accelerated Gradient Descent Methods for Convex Optimization

TL;DR

The paper introduces A$^2$GD, an adaptive accelerated gradient method for smooth and composite convex optimization that uses Lyapunov-based analysis to adapt local smoothness $L_k$ and strong convexity $\mu_k$, triggering line search only when accumulated perturbations become positive. By integrating adaptive step sizes with momentum acceleration and a line-search–reduced scheme, A$^2$GD achieves accelerated convergence with fewer gradient evaluations. Theoretical results establish accelerated linear convergence for smooth and composite convex objectives, while extensive numerical experiments—on regularized logistic regression, MLE of the information matrix, and $\ell_{1-2}$ nonconvex problems—demonstrate substantial practical gains and robustness to hyperparameters. The approach offers a principled balance between adaptivity and acceleration, with potential extensions to stochastic settings and nonconvex regimes.

Abstract

This work proposes A$^2$GD, a novel adaptive accelerated gradient descent method for convex and composite optimization. Smoothness and convexity constants are updated via Lyapunov analysis. Inspired by stability analysis in ODE solvers, the method triggers line search only when accumulated perturbations become positive, thereby reducing gradient evaluations while preserving strong convergence guarantees. By integrating adaptive step size and momentum acceleration, A$^2$GD outperforms existing first-order methods across a range of problem settings.

Figures (19)

• Figure 1: The accumulated perturbation $p_k$ (green) stays negative and approaches zero. The Lyapunov values $E_k$ (blue) decay faster than the theoretical exponential rate $\left(\prod_{i=0}^{k}\delta_i\right)E_0$ (black). In the early iterations, $E_k$ decreases even more rapidly due to the large negative term $b_k^{(2)} = -\tfrac{1}{2L_k}\|\nabla f(x_k)\|^2$.
• Figure 2: For a logistic regression problem (\ref{['eq:logreg']}), gradient descent with line search enforcing $b_k^{(1)} \le 0$ (top curve) triggers backtracking every $3$--$4$ iterations on average. In contrast, ad-GD, which performs line search only when $p_k>0$, requires far fewer activations (bottom curve). Red dots mark iterations where line search is triggered.
• Figure 3: Comparison without restarting.
• Figure 4: Comparison with restarting.
• Figure 5: Comparison with other adaptive methods.

Theorems & Definitions (21)

• Lemma 2.1: A variant of Lemma 5.7.1. in Gautschi:2011Numerical
• Theorem 2.2
• proof
• Remark 2.1
• Remark 2.2
• Lemma 3.1
• Theorem 3.2
• Proposition 4.1
• proof
• Lemma 4.2