Accelerated Inertial Gradient Algorithms with Vanishing Tikhonov Regularization

TL;DR

This paper develops TRIGA, an explicit inertial gradient method augmented with a vanishing Tikhonov regularization term for smooth convex minimization with Lipschitz gradients. Through a discrete Lyapunov analysis of an associated second-order system, TRIGA achieves accelerated convergence of objective values while ensuring strong convergence of the iterates to the minimum-norm solution $x^*$ for polynomial decays $\varepsilon_k = k^{-p}$ with $0<p<2$ (and fast rates at the critical case $p=2$ without guaranteed strong convergence). The authors provide a thorough convergence theory for general $\varepsilon_k$ schedules and give detailed rate results for specific choices, complemented by numerical experiments on synthetic, benchmark, and real data demonstrating practical performance and robustness. The work advances the design of explicit, accelerated schemes that simultaneously optimize convergence rates and solution selection, with implications for large-scale convex optimization and learning tasks.

Abstract

In this paper, we study an explicit Tikhonov-regularized inertial gradient algorithm for smooth convex minimization with Lipschitz continuous gradient. The method is derived via an explicit time discretization of a damped inertial system with vanishing Tikhonov regularization. Under appropriate control of the decay rate of the Tikhonov term, we establish accelerated convergence of the objective values to the minimum together with strong convergence of the iterates to the minimum-norm minimizer. In particular, for polynomial schedules $\varepsilon_k = k^{-p}$ with $0<p<2$, we prove strong convergence to the minimum-norm solution while preserving fast objective decay. In the critical case $p=2$, we still obtain fast rates for the objective values, while our analysis does not guarantee strong convergence to the minimum-norm minimizer. Furthermore, we provide a thorough theoretical analysis for several choices of Tikhonov schedules. Numerical experiments on synthetic, benchmark, and real datasets illustrate the practical performance of the proposed algorithm.

Figures (8)

• Figure 1: Strong convergence to the minimum-norm solution $x^*=(0.5,0.5)$ in $\mathbb{R}^2$
• Figure 2: Performance in iterations of \ref{['algo:nadtr']} (left) and \ref{['algo:triga']} (right) with $s = \dfrac{1}{1.1L}$ in terms of the criteria (from top to bottom): $f(x_k) - \min_\mathcal{H} f$, $\Vert \nabla f(x_k) \Vert$, $\Vert x_k - x^* \Vert$ and $\Vert x_k - x_{k-1}\Vert$
• Figure 3: Performance in iterations of \ref{['algo:nadtr']} (left) and \ref{['algo:triga']} (right) with $s = \dfrac{1}{2.1 L}$ in terms of the criteria (from top to bottom): $f(x_k) - \min_\mathcal{H} f$, $\Vert \nabla f(x_k) \Vert$, $\Vert x_k - x^* \Vert$ and $\Vert x_k - x_{k-1}\Vert$
• Figure 4: Performance profiles of \ref{['algo:triga']} for different values of $p$ when $s = \frac{1}{1.1L}$ (first row) and $s = \frac{1}{2.1L}$ (second row) in terms of CPU time (in seconds) (left) and number of iterations (right) on benchmark datasets of linear least-squares problems
• Figure 5: Performance profiles of \ref{['algo:nadtr']} for different values of $p$ when $s = \frac{1}{1.1L}$ (first row) and $s = \frac{1}{2.1L}$ (second row) in terms of CPU time (in seconds) (left) and number of iterations (right) on benchmark datasets of linear least-squares problems

Theorems & Definitions (25)

• Lemma 2.1: Attouch:sjo:96Attouch:amo:23
• Lemma 2.2: Lemma 4, Attouch:amo:23
• Lemma 2.3
• proof
• Definition 2.4
• Theorem 2.5
• proof
• Proposition 2.6
• proof
• Remark 2.7