Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast convex optimization via closed-loop time scaling of gradient dynamics

Hedy Attouch, Radu Ioan Bot, Dang-Khoa Nguyen

TL;DR

The paper addresses fast convex optimization by designing adaptive accelerated gradient methods whose damping is controlled in a closed-loop fashion. It introduces time-scaling and averaging to produce autonomous damped inertial dynamics with vanishing viscous damping and Hessian-driven damping, yielding fast convergence of objective values, gradient decay, and convergence to minimizers; these continuous Dynamics inspire proximal discretizations and adaptive step-size rules. The authors derive general time-scale results, specialized to velocity- and gradient-based damping, and compare with Lin–Jordan, demonstrating improved convergence rates such as $f(x(t))-\,\inf f = o(1/t^{2p})$ for certain parameters, while keeping trajectory convergence guarantees. They then develop proximal-explicit discretizations (PEAS) and inertial proximal algorithms with adaptive stepsizes, provide a geometric interpretation, extend to nonsmooth settings via Moreau envelopes, and validate the approach with numerical experiments that show competitive performance and reduced oscillations. Overall, the work provides a flexible, theoretically grounded framework for autonomous, adaptive acceleration in convex optimization with potential extensions to composite, monotone, and stochastic settings.

Abstract

In a Hilbert setting, for convex differentiable optimization, we develop a general framework for adaptive accelerated gradient methods. They are based on damped inertial dynamics where the coefficients are designed in a closed-loop way. Specifically, the damping is a feedback control of the velocity, or of the gradient of the objective function. For this, we develop a closed-loop version of the time scaling and averaging technique introduced by the authors. We thus obtain autonomous inertial dynamics which involve vanishing viscous damping and implicit Hessian driven damping. By simply using the convergence rates for the continuous steepest descent and Jensen's inequality, without the need for further Lyapunov analysis, we show that the trajectories have several remarkable properties at once: they ensure fast convergence of values, fast convergence of the gradients towards zero, and they converge to optimal solutions. Our approach leads to parallel algorithmic results, that we study in the case of proximal algorithms. These are among the very first general results of this type obtained using autonomous dynamics. Since the proposed numerical methods are based on proximal techniques, the results can be extended to a broader class, specifically to the problem of minimizing a proper, lower semicontinuous, and convex function. Numerical experiments are conducted to demonstrate the efficiency of the proposed methods.

Fast convex optimization via closed-loop time scaling of gradient dynamics

TL;DR

The paper addresses fast convex optimization by designing adaptive accelerated gradient methods whose damping is controlled in a closed-loop fashion. It introduces time-scaling and averaging to produce autonomous damped inertial dynamics with vanishing viscous damping and Hessian-driven damping, yielding fast convergence of objective values, gradient decay, and convergence to minimizers; these continuous Dynamics inspire proximal discretizations and adaptive step-size rules. The authors derive general time-scale results, specialized to velocity- and gradient-based damping, and compare with Lin–Jordan, demonstrating improved convergence rates such as for certain parameters, while keeping trajectory convergence guarantees. They then develop proximal-explicit discretizations (PEAS) and inertial proximal algorithms with adaptive stepsizes, provide a geometric interpretation, extend to nonsmooth settings via Moreau envelopes, and validate the approach with numerical experiments that show competitive performance and reduced oscillations. Overall, the work provides a flexible, theoretically grounded framework for autonomous, adaptive acceleration in convex optimization with potential extensions to composite, monotone, and stochastic settings.

Abstract

In a Hilbert setting, for convex differentiable optimization, we develop a general framework for adaptive accelerated gradient methods. They are based on damped inertial dynamics where the coefficients are designed in a closed-loop way. Specifically, the damping is a feedback control of the velocity, or of the gradient of the objective function. For this, we develop a closed-loop version of the time scaling and averaging technique introduced by the authors. We thus obtain autonomous inertial dynamics which involve vanishing viscous damping and implicit Hessian driven damping. By simply using the convergence rates for the continuous steepest descent and Jensen's inequality, without the need for further Lyapunov analysis, we show that the trajectories have several remarkable properties at once: they ensure fast convergence of values, fast convergence of the gradients towards zero, and they converge to optimal solutions. Our approach leads to parallel algorithmic results, that we study in the case of proximal algorithms. These are among the very first general results of this type obtained using autonomous dynamics. Since the proposed numerical methods are based on proximal techniques, the results can be extended to a broader class, specifically to the problem of minimizing a proper, lower semicontinuous, and convex function. Numerical experiments are conducted to demonstrate the efficiency of the proposed methods.
Paper Structure (28 sections, 16 theorems, 197 equations, 5 figures, 4 algorithms)

This paper contains 28 sections, 16 theorems, 197 equations, 5 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Time scale and averaging: the open-loop approach
  3. Closed-loop control
  4. Link with the existing literature
  5. Organization of the paper
  6. Closed-loop time scaling of the steepest descent
  7. Formulation of the closed-loop time scaling
  8. Lower bound estimate of the time scaling $\tau(t)$
  9. Closed-loop control of (SD) via the velocity
  10. Closed-loop control of (SD) via the norm of gradient
  11. Accelerated gradient systems with closed-loop control of the damping
  12. General results concerning time scale and averaging
  13. Damped inertial system via closed-loop control of the velocity
  14. Damped inertial system via closed-loop control of the gradient
  15. Comparison with the Lin-Jordan approach
  16. ...and 13 more sections

Key Result

theorem 1

Suppose that $f \colon {\mathcal{H}} \to {\mathbb R}$ satisfies $(\mathcal{A})$. Let us choose the positive parameters according to $q >0$, $p \geq 1$, and $\gamma >1$. Let $x \colon \left[ t_{0} , + \infty \right[ \to {\mathcal{H}}$ be a solution trajectory of the following system Then we have the fast convergence of values: as $t\to +\infty$ Moreover, the solution trajectory $x(t)$ converges we

Figures (5)

  • Figure 1: Numerical comparisons between PPA, FISTA, and Algorithm \ref{['algo:prox-vbased']} and Algorithm \ref{['algo:prox-gbased']}.
  • Figure 2: Numerical comparisons between PPA, FISTA, and Algorithm \ref{['algo:prox-vbased']} and Algorithm \ref{['algo:prox-gbased']} for $\left( m , n \right) := \left( 1 , 10^{2} \right)$.
  • Figure 3: Numerical comparisons between PPA, FISTA, and Algorithm \ref{['algo:prox-vbased']} and Algorithm \ref{['algo:prox-gbased']} for $\left( m , n \right) := \left( 10^{2} , 10^{2} \right)$.
  • Figure 4: Numerical comparisons between PPA, FISTA, and Algorithm \ref{['algo:prox-vbased']} and Algorithm \ref{['algo:prox-gbased']} for $\left( m , n \right) := \left( 10^{4} , 10^{2} \right)$.
  • Figure :

Theorems & Definitions (31)

  • theorem 1
  • proposition thmcounterproposition
  • proof
  • lemma thmcounterlemma
  • proof
  • theorem 2
  • proof
  • theorem 3
  • proof
  • remark thmcounterremark
  • ...and 21 more