Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound

TL;DR

The paper addresses nonasymptotic convergence of accelerated first-order methods under inexact gradient oracles with an absolute error bound in deterministic convex optimization. It leverages the Performance Estimation Problem (PEP) framework to derive explicit nonasymptotic bounds for iGOGM and iGFGM, revealing a two-component bound: a diminishing term and an accumulated error term that is independent of the initial condition. The authors show how to balance rate and error via a rate-error tradeoff, and they formulate an optimal inexactness schedule that minimizes oracle cost while preserving acceleration. They also extend PEP to inexact settings, providing analytical dual solutions that back the main results and offering practical guidance on how to allocate gradient-estimation effort across iterations. Overall, the work provides a principled way to design and tune inexact-gradient accelerated methods for deterministic convex problems, with clear implications for bilevel, composition, and zero-/Gaussian-smoothed gradient contexts.

Abstract

Performance analysis of first-order algorithms with inexact oracles has gained recent attention due to various emerging applications in which obtaining exact gradients is impossible or computationally expensive. Previous research has demonstrated that the performance of accelerated first-order methods is more sensitive to gradient errors compared with non-accelerated ones. This paper investigates the nonasymptotic convergence bound of two accelerated methods with inexact gradients to solve deterministic smooth convex problems. Performance Estimation Problem (PEP) is used as the primary tool to analyze the convergence bounds of the underlying algorithms. By finding an analytical solution to PEP, we derive novel convergence bounds of Generalized Optimized Gradient Method (GOGM) and Generalized Fast Gradient Method (GFGM) with inexact gradient oracles following the absolute error bound. The derived bounds allow varying oracle inexactness along the iterations; furthermore, their accumulated error terms are independent of the initial condition and any unknown parameters. Furthermore, we analyze the tradeoff between the vanishing term and the accumulated error in the convergence bound that guides finding the optimal stepsize. Finally, we determine the optimal strategy to set the gradient inexactness along iterations (if possible in a given application), ensuring that the accumulated error remains subordinate to the vanishing term.

Figures (7)

• Figure 1: Values of $u_k$ for OGM-$a$ with different values of $a$ and $K$.
• Figure 2: Convergence rate and accumulated error for OGM-$a$ for different $a$ and $K$ values.
• Figure 3: Total $\eta$-complexity of iOGM-4 for $h(\eta)$ with power law decay with $c_1=1$ and $L=1, R=1$.
• Figure 4: Total $\eta$-complexity of iOGM-4 for $h(\eta)$ with exponential decay with $q_1=1,q_2=e$ and $L=1,R=1$.
• Figure 5: Difference between the numerical solution and the analytical solution (for the problem with exact oracle), i.e., \ref{['eq:OGM-solution']}, for $\mathbf{v}$. Solid lines are the average of 10 instances and the shaded areas represent the variation of individual instances.

Theorems & Definitions (21)

• Theorem 2.2
• Remark 2.3
• Theorem 2.4
• Proposition 2.5
• Lemma 2.6: $h(\eta)$ with power law decay
• proof
• Lemma 2.7: $h(\eta)$ with exponential decay
• proof
• Definition 3.1: $\mathcal{F}_{\mu,L}$-interpolation (Definition 2 in Taylor2016SmoothStronglyConvex)
• Theorem 3.2: $\mathcal{F}_{\mu,L}$-interpolable (Theorem 4 in Taylor2016SmoothStronglyConvex)