A Family of Controllable Momentum Coefficients for Forward-Backward Accelerated Algorithms
Mingwei Fu, Bin Shi
TL;DR
This work introduces a family of controllable momentum coefficients for forward-backward accelerated methods, centered on an $\alpha$-th power momentum form with adaptive $r$ at the critical step size $s=1/L$. By designing a Lyapunov function that excludes kinetic energy and expresses energies in terms of $x_k$ and $y_k$, the authors prove a controllable $O\left(1/k^{2\alpha}\right)$ convergence for NAG-$\alpha$ when $r>2\alpha$, and extend this rate to monotone variants and proximal algorithms, including FISTA-$\alpha$ and M-FISTA-$\alpha$. At the critical step size, tuning $r$ according to $\alpha$ yields inverse-polynomial convergence of arbitrary degree, offering a tunable acceleration mechanism for a broad class of smooth and composite problems. The results bridge the gap between classical acceleration and proximal forward-backward methods, with implications for both theory and practical optimization, though the analysis relies on strong convexity and leaves open the exploration of weaker conditions.
Abstract
Nesterov's accelerated gradient method (NAG) marks a pivotal advancement in gradient-based optimization, achieving faster convergence compared to the vanilla gradient descent method for convex functions. However, its algorithmic complexity when applied to strongly convex functions remains unknown, as noted in the comprehensive review by Chambolle and Pock [2016]. This issue, aside from the critical step size, was addressed by Li et al. [2024b], with the monotonic case further explored by Fu and Shi [2024]. In this paper, we introduce a family of controllable momentum coefficients for forward-backward accelerated methods, focusing on the critical step size $s=1/L$. Unlike traditional linear forms, the proposed momentum coefficients follow an $α$-th power structure, where the parameter $r$ is adaptively tuned to $α$. Using a Lyapunov function specifically designed for $α$, we establish a controllable $O\left(1/k^{2α} \right)$ convergence rate for the NAG-$α$ method, provided that $r > 2α$. At the critical step size, NAG-$α$ achieves an inverse polynomial convergence rate of arbitrary degree by adjusting $r$ according to $α> 0$. We further simplify the Lyapunov function by expressing it in terms of the iterative sequences $x_k$ and $y_k$, eliminating the need for phase-space representations. This simplification enables us to extend the controllable $O \left(1/k^{2α} \right)$ rate to the monotonic variant, M-NAG-$α$, thereby enhancing optimization efficiency. Finally, by leveraging the fundamental inequality for composite functions, we extended the controllable $O\left(1/k^{2α} \right)$ rate to proximal algorithms, including the fast iterative shrinkage-thresholding algorithm (FISTA-$α$) and its monotonic counterpart (M-FISTA-$α$).