Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-optimal Closed-loop Method via Lyapunov Damping for Convex Optimization

Severin Maier, Camille Castera, Peter Ochs

TL;DR

This work introduces an autonomous system with closed-loop damping for first-order convex optimization by coupling the damping and the speed of convergence of the system via a well-chosen Lyapunov function and derives an algorithm from it.

Abstract

We introduce an autonomous system with closed-loop damping for first-order convex optimization. While, to this day, optimal rates of convergence are almost exclusively achieved by non-autonomous methods via open-loop damping (e.g., Nesterov's algorithm), we show that our system, featuring a closed-loop damping, exhibits a rate arbitrarily close to the optimal one. We do so by coupling the damping and the speed of convergence of the system via a well-chosen Lyapunov function. By discretizing our system we then derive an algorithm and present numerical experiments supporting our theoretical findings.

Near-optimal Closed-loop Method via Lyapunov Damping for Convex Optimization

TL;DR

This work introduces an autonomous system with closed-loop damping for first-order convex optimization by coupling the damping and the speed of convergence of the system via a well-chosen Lyapunov function and derives an algorithm from it.

Abstract

We introduce an autonomous system with closed-loop damping for first-order convex optimization. While, to this day, optimal rates of convergence are almost exclusively achieved by non-autonomous methods via open-loop damping (e.g., Nesterov's algorithm), we show that our system, featuring a closed-loop damping, exhibits a rate arbitrarily close to the optimal one. We do so by coupling the damping and the speed of convergence of the system via a well-chosen Lyapunov function. By discretizing our system we then derive an algorithm and present numerical experiments supporting our theoretical findings.
Paper Structure (22 sections, 13 theorems, 73 equations, 5 figures, 1 algorithm)

This paper contains 22 sections, 13 theorems, 73 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem setting
  3. Our contribution
  4. Organization
  5. Related Work
  6. ODEs for optimization
  7. Closed-loop damping
  8. Open-loop damping and proof techniques
  9. Preliminaries and Existence of Solutions
  10. Main Result: Convergence Rates for LD
  11. The Lyapunov function of LD
  12. Preliminary convergence results
  13. Rates of convergence for E
  14. Drawbacks of the approach
  15. Algorithmic case and Numerical Experiments
  16. ...and 7 more sections

Key Result

Theorem 1.1

Assume that $f$ is a continuously differentiable convex function and that $\mathop{\mathrm{\arg\!\min}}\limits_{\mathcal{H}} f \neq \emptyset$. Then, for any bounded solution $x$ of LD and for any $\delta > 0$,

Figures (5)

  • Figure 1: Comparison of our autonomous system \ref{['LD']} and the non-autonomous one \ref{['eq::AVD']} for different initial times $t_0$ on the 2D function $f(x_1,x_2) = x_1^4+0.1x_2^4$. The left plot shows the evolution of the function values over time. The right plot shows the trajectories in the space $(x_1,x_2)$. Different initial times heavily affect the solution of \ref{['eq::AVD']}, but not \ref{['LD']}. We approximated the solutions using NAG for \ref{['eq::AVD']} and LYDIA (see Algorithm \ref{['algo::LD']}) for \ref{['LD']}, both with very small step-sizes.
  • Figure 2: Rate of convergence of LD, AVD, HBF and GD on $f_\text{flat}$. Left: Evolution of the function values. Right: Evolution of the Lyapunov function.
  • Figure 3: Rate of convergence of LD, AVD, HBF and GD on $f_\text{non-KL}$. Left: Evolution of the function values. Right: Evolution of the Lyapunov function.
  • Figure 4: Rate of convergence of LD, AVD, HBF and GD on $f_\text{uneven}$. Left: Evolution of the function values. Right: Evolution of the Lyapunov function.
  • Figure 5: Rate of convergence of LD, AVD, HBF and GD on $f_\text{contmin}$. Left: Evolution of the function values. Right: Evolution of the Lyapunov function.

Theorems & Definitions (33)

  • Theorem 1.1
  • Definition 1
  • Theorem 3.1
  • Remark 3.2
  • Definition 2
  • Lemma 4.1
  • proof
  • Remark 4.2
  • Theorem 4.3
  • Lemma 4.4
  • ...and 23 more