Adaptive Inertial Method

TL;DR

The paper addresses accelerating first-order methods for unconstrained convex optimization with Lipschitz gradients by introducing the Adaptive Inertial Method (AIM), a flexible framework that uses a customizable inertial term and an adaptive γ_k to ensure descent. AIM unifies and extends acceleration techniques by enabling adaptive parameter selection and a proximal-point interpretation with a non-Euclidean metric, achieving a guaranteed global rate of $O(1/k)$ under mild assumptions, and connecting to quasi-Newton and regularized Newton methods under specific inertial constructions to reach $O(1/k^2)$. The main contributions include (i) a rigorous convergence analysis with a descent framework for adaptive inertial weights, (ii) a spectrum of inertial-term options (physical, quasi-Newton, and Hessian-gradient), and (iii) extensive numerical experiments showing AIM, particularly AIM_Hg, outperforming several baselines on logistic regression and $\mathcal{L}_2$-$\mathcal{L}_p$ minimization. This work provides a practical, tunable acceleration framework that can incorporate second-order information without Hessian inversions, potentially impacting large-scale optimization in machine learning and data analysis. Overall, AIM offers a versatile approach to accelerate convex optimization with adaptive, theory-backed guarantees and strong empirical performance.

Abstract

In this paper, we introduce the Adaptive Inertial Method (AIM), a novel framework for accelerated first-order methods through a customizable inertial term. We provide a rigorous convergence analysis establishing a global convergence rate of O(1/k) under mild conditions, requiring only convexity and local Lipschitz differentiability of the objective function. Our method enables adaptive parameter selection for the inertial term without manual tuning. Furthermore, we derive the particular form of the inertial term that transforms AIM into a new Quasi-Newton method. Notably, under specific circumstances, AIM coincides with the regularized Newton method, achieving an accelerated rate of O(1/k^2) without Hessian inversions. Through extensive numerical experiments, we demonstrate that AIM exhibits superior performance across diverse optimization problems, highlighting its practical effectiveness.

Figures (3)

• Figure 1: Choose $\bm{m}^k = \bm{m}_v^k$ and $\mu_k = 0.5$
• Figure 1: Convergence behavior of AIM and selected benchmark algorithms for logistic regression with $\mathcal{L}_2$ regularization under varying regularization strengths. The convergence trajectory of AIM with Hessian-gradient inertia (AIM_Hg) exhibits minor oscillations but closely approximates that of DRSOM, which achieves the $O(1/k^2)$ convergence rate for convex problems.
• Figure 2: Convergence behavior of AIM and selected benchmark algorithms for $\mathcal{L}_2-\mathcal{L}_p$ minimization. AIM with Hessian-gradient inertia (AIM_Hg) consistently outperforms all other methods. While exhibiting slightly more oscillatory behavior than DRSOM, AIM_Hg achieves steeper descent trajectories.

Theorems & Definitions (19)

• Remark 2.1
• Remark 2.2
• Remark 2.3
• Remark 2.4
• Remark 2.5
• Lemma 2.6
• Proof 1
• Remark 2.7
• Lemma 2.8
• Proof 2