Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards faster first order methods: A continuous-time model to interpolate between speed and function value restart

Juan José Maulén, Huiyuan Guo, Juan Peypouquet

Abstract

We introduce a new restarting scheme for a continuous inertial dynamics with Hessian driven-damping, and establish a linear convergence rate for the function values along the restarted trajectories. The proposed routine is implemented without knowing the strong convexity parameter, and is a generalization of existing speed restart schemes. It interpolates between speed and function value restarts, considerably delaying the restarting time, while preserving convergence and function value decrease. Numerical experiments show an improvement in the convergence rates for both continuous-time dynamical systems, and the associated accelerated first-order algorithms derived via time discretization.

Towards faster first order methods: A continuous-time model to interpolate between speed and function value restart

Abstract

We introduce a new restarting scheme for a continuous inertial dynamics with Hessian driven-damping, and establish a linear convergence rate for the function values along the restarted trajectories. The proposed routine is implemented without knowing the strong convexity parameter, and is a generalization of existing speed restart schemes. It interpolates between speed and function value restarts, considerably delaying the restarting time, while preserving convergence and function value decrease. Numerical experiments show an improvement in the convergence rates for both continuous-time dynamical systems, and the associated accelerated first-order algorithms derived via time discretization.

Paper Structure

This paper contains 16 sections, 8 theorems, 69 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The solutions of \ref{['eq:din_avd']}
  3. Restarting schemes
  4. From restarting schemes to linear convergence
  5. An extended speed restart for \ref{['eq:din_avd']}
  6. Lower bound for the restarting time
  7. A few remarks on the values of $\tau_3$
  8. Upper bound for the restarting time
  9. Uniform function value decrease and conclusion
  10. Numerical Illustrations
  11. Continuous case
  12. A proposed heuristic for a discrete criterion
  13. Illustrative examples
  14. Kernel Regularized Learning
  15. Conclusion
  16. ...and 1 more sections

Key Result

Proposition 1

For every $t \in (0,\tau_1)$, we have Moreover, for every $t \in (0,\tau_2)$, we have $\blacktriangleleft$$\blacktriangleleft$

Figures (6)

  • Figure 1: Function values along the solutions of \ref{['eq:din_avd']} for the function $\phi$ in Example \ref{['ex:intr']}, with $\alpha=3$, $\beta=1$, initial condition $x(1)=(1,1,1)$ and $\dot{x}(1)=0$, on the interval $[1,35]$. For \ref{['eq:hr_dinavd']}, we consider $\gamma=1$ and $r=\frac{\alpha\beta}{2}$ to fulfill the hypotheses in wang2025fast.
  • Figure 2: In blue-green, we show the graph of $\varphi_{z,\lambda}(t)$ in the context of Example \ref{['ex:intr']}, with $\rho=1$, $z=(1,1,1)$, $\alpha=3$. The function values $\phi(x_z(t))$ are displayed in red.
  • Figure 3: Values along the trajectory, for the solutions of \ref{['eq:din_avd']} with $\phi$ a quadratic function as in \ref{['eq:func_cont']}, for different choices of $\lambda$.
  • Figure 4: Values of $\phi(x_{k})-\phi^{*}$ along the iterations for non restart, speed restart scheme ($\lambda=0$) and extended restart scheme with different choices of $\lambda$.
  • Figure 5: Values of $\phi(x_{k})-\phi^{*}$ along the iterations for non restart, speed restart scheme ($\lambda=0$), extended speed restart and the versions with the warm start.
  • ...and 1 more figures

Theorems & Definitions (15)

  • Example 1
  • Proposition 1
  • Lemma 1
  • proof
  • Theorem 1
  • Lemma 2
  • proof
  • Corollary 1
  • Proposition 2
  • proof
  • ...and 5 more