Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heavy Ball Momentum for Non-Strongly Convex Optimization

Jean-François Aujol, Charles Dossal, Hippolyte Labarrière, Aude Rondepierre

TL;DR

The paper extends heavy-ball type inertial methods to convex composite optimization under a quadratic growth condition $\,\mathcal{G}^2_\mu$ without requiring uniqueness of the minimizer. It shows that properly parameterized HB variants, notably a V-FISTA-like scheme with inertia tuned to $\alpha=1-\omega\sqrt{\kappa}$, achieve fast exponential decay in the objective gap $F(x_n)-F^*$ with rate depending on $\sqrt{\kappa}$, outperforming restarted FISTA in this setting. Complementing the discrete results, the authors analyze the Heavy Ball ODE and prove fast exponential decay for trajectories even when multiple minimizers exist, under second-order regularity of the minimizer set. The work also provides robustness to misestimation of the growth parameter and presents comprehensive proofs via Lyapunov energies that connect discrete algorithms with their continuous-time dynamics. Overall, the results significantly broaden the practical and theoretical reach of inertial methods for non-strongly convex problems with growth conditions, enabling reliable fast convergence without requiring minimizer uniqueness.

Abstract

When considering the minimization of a quadratic or strongly convex function, it is well known that first-order methods involving an inertial term weighted by a constant-in-time parameter are particularly efficient (see Polyak [32], Nesterov [28], and references therein). By setting the inertial parameter according to the condition number of the objective function, these methods guarantee a fast exponential decay of the error. We prove that this type of schemes (which are later called Heavy Ball schemes) is relevant in a relaxed setting, i.e. for composite functions satisfying a quadratic growth condition. In particular, we adapt V-FISTA, introduced by Beck in [10] for strongly convex functions, to this broader class of functions. To the authors' knowledge, the resulting worst-case convergence rates are faster than any other in the literature, including those of FISTA restart schemes. No assumption on the set of minimizers is required and guarantees are also given in the non-optimal case, i.e. when the condition number is not exactly known. This analysis follows the study of the corresponding continuous-time dynamical system (Heavy Ball with friction system), for which new convergence results of the trajectory are shown.

Heavy Ball Momentum for Non-Strongly Convex Optimization

TL;DR

The paper extends heavy-ball type inertial methods to convex composite optimization under a quadratic growth condition without requiring uniqueness of the minimizer. It shows that properly parameterized HB variants, notably a V-FISTA-like scheme with inertia tuned to , achieve fast exponential decay in the objective gap with rate depending on , outperforming restarted FISTA in this setting. Complementing the discrete results, the authors analyze the Heavy Ball ODE and prove fast exponential decay for trajectories even when multiple minimizers exist, under second-order regularity of the minimizer set. The work also provides robustness to misestimation of the growth parameter and presents comprehensive proofs via Lyapunov energies that connect discrete algorithms with their continuous-time dynamics. Overall, the results significantly broaden the practical and theoretical reach of inertial methods for non-strongly convex problems with growth conditions, enabling reliable fast convergence without requiring minimizer uniqueness.

Abstract

When considering the minimization of a quadratic or strongly convex function, it is well known that first-order methods involving an inertial term weighted by a constant-in-time parameter are particularly efficient (see Polyak [32], Nesterov [28], and references therein). By setting the inertial parameter according to the condition number of the objective function, these methods guarantee a fast exponential decay of the error. We prove that this type of schemes (which are later called Heavy Ball schemes) is relevant in a relaxed setting, i.e. for composite functions satisfying a quadratic growth condition. In particular, we adapt V-FISTA, introduced by Beck in [10] for strongly convex functions, to this broader class of functions. To the authors' knowledge, the resulting worst-case convergence rates are faster than any other in the literature, including those of FISTA restart schemes. No assumption on the set of minimizers is required and guarantees are also given in the non-optimal case, i.e. when the condition number is not exactly known. This analysis follows the study of the corresponding continuous-time dynamical system (Heavy Ball with friction system), for which new convergence results of the trajectory are shown.
Paper Structure (35 sections, 12 theorems, 147 equations, 3 figures, 3 tables)

This paper contains 35 sections, 12 theorems, 147 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Geometry of convex functions and inertial algorithms: definitions and state of the art.
  3. Framework and notations
  4. Geometry assumptions.
  5. Inertial methods.
  6. Convergence rate of inertial algorithms under quadratic growth assumptions
  7. Forward-Backward.
  8. Nesterov's acceleration.
  9. Restarting strategies.
  10. Heavy Ball schemes.
  11. Fast exponential decay for Heavy Ball type algorithms
  12. Fast exponential decay of the error.
  13. Parametric convergence rates.
  14. Guarantees in suboptimal cases.
  15. Convergence rates for the trajectories of the Heavy Ball dynamical system
  16. ...and 20 more sections

Key Result

Theorem 1

Let $F=f+h$ where $f$ is a convex differentiable function having a $L$-Lipschitz gradient for some $L>0$, and $h$ a proper convex l.s.c. function. Assume that $F$ satisfies a quadratic growth condition $\mathcal{G}_\mu^2$ for some real parameter $\mu>0$. Let $(x_n)_{n\in\mathbb{N}}$ be the sequence and

Figures (3)

  • Figure 1: Evolution of $\tau$ with respect to $\omega$ for several values of $\kappa$ such that $(\tau,\omega,\kappa)$ satisfy \ref{['eq:cond_vfista_eg']}.
  • Figure 2: Behaviour of $\dot{x^*}$ for a set of minimizers having a $C^2$ bound (on the left) and a polyhedral set of minimizers (on the right).
  • Figure 3: Comparison of the highest rate $\tau$ satisfying \ref{['eq:cond_vfista_eg']} for $\kappa=\frac{1}{10}$ and $\omega>0$ with the function $\omega\mapsto\frac{2}{3\omega}$.

Theorems & Definitions (18)

  • Definition 1: Strong convexity $\mathcal{S}_\mu$
  • Definition 2: Quadratic growth condition $\mathcal{G}^2_\mu$
  • Definition 3
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Corollary 2
  • Theorem 3
  • Proposition 1
  • Remark 1
  • ...and 8 more