Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving convex optimization problems via a second order dynamical system with implicit Hessian damping and Tikhonov regularization

Szilárd Csaba László

TL;DR

The paper studies a convex optimization problem via a continuous-time second-order dynamical system with a time-dependent damping and a Tikhonov regularization term, incorporating implicit Hessian damping through a shifted gradient. By establishing Lyapunov-based energy estimates and careful parameter regimes, it proves both weak convergence to minimizers and, under stronger assumptions, strong convergence to the minimal-norm minimizer, along with fast rates for the objective gap and decay of velocities. The key contribution is showing how the damping and regularization parameters interact to yield fast convergence of function values while enabling either weak or strong convergence, and deriving integral estimates that underpin the convergence rates. This work also connects continuous dynamics with inertial gradient-type algorithms obtained via explicit discretization, extending prior results to the case $0<q<1$ and providing conditions under which strong convergence is guaranteed with practical rate guarantees.

Abstract

This paper deals with a second order dynamical system with a Tikhonov regularization term in connection to the minimization problem of a convex Fréchet differentiable function. The fact that beside the asymptotically vanishing damping we also consider an implicit Hessian driven damping in the dynamical system under study allows us, via straightforward explicit discretization, to obtain inertial algorithms of gradient type. We show that the value of the objective function in a generated trajectory converges rapidly to the global minimum of the objective function and depending the Tikhonov regularization parameter the generated trajectory converges weakly to a minimizer of the objective function or the generated trajectory converges strongly to the element of minimal norm from the $\argmin$ set of the objective function. We also obtain the fast convergence of the velocities towards zero and some integral estimates. Our analysis reveals that the Tikhonov regularization parameter and the damping parameters are strongly correlated, there is a setting of the parameters that separates the cases when weak convergence of the trajectories to a minimizer and strong convergence of the trajectories to the minimal norm minimizer can be obtained.

Solving convex optimization problems via a second order dynamical system with implicit Hessian damping and Tikhonov regularization

TL;DR

The paper studies a convex optimization problem via a continuous-time second-order dynamical system with a time-dependent damping and a Tikhonov regularization term, incorporating implicit Hessian damping through a shifted gradient. By establishing Lyapunov-based energy estimates and careful parameter regimes, it proves both weak convergence to minimizers and, under stronger assumptions, strong convergence to the minimal-norm minimizer, along with fast rates for the objective gap and decay of velocities. The key contribution is showing how the damping and regularization parameters interact to yield fast convergence of function values while enabling either weak or strong convergence, and deriving integral estimates that underpin the convergence rates. This work also connects continuous dynamics with inertial gradient-type algorithms obtained via explicit discretization, extending prior results to the case and providing conditions under which strong convergence is guaranteed with practical rate guarantees.

Abstract

This paper deals with a second order dynamical system with a Tikhonov regularization term in connection to the minimization problem of a convex Fréchet differentiable function. The fact that beside the asymptotically vanishing damping we also consider an implicit Hessian driven damping in the dynamical system under study allows us, via straightforward explicit discretization, to obtain inertial algorithms of gradient type. We show that the value of the objective function in a generated trajectory converges rapidly to the global minimum of the objective function and depending the Tikhonov regularization parameter the generated trajectory converges weakly to a minimizer of the objective function or the generated trajectory converges strongly to the element of minimal norm from the set of the objective function. We also obtain the fast convergence of the velocities towards zero and some integral estimates. Our analysis reveals that the Tikhonov regularization parameter and the damping parameters are strongly correlated, there is a setting of the parameters that separates the cases when weak convergence of the trajectories to a minimizer and strong convergence of the trajectories to the minimal norm minimizer can be obtained.
Paper Structure (10 sections, 13 theorems, 143 equations)

This paper contains 10 sections, 13 theorems, 143 equations.

Table of Contents

  1. Introduction
  2. The problem formulation and a model result
  3. Motivation and related works
  4. The organization of the paper
  5. Existence and uniqueness of the trajectories generated by the dynamical system \ref{['DynSys']}
  6. Weak convergence for the trajectories of the regularized dynamical system \ref{['DynSys']} and fast convergence for the function values in a generated trajectory
  7. On the strong convergence of the trajectories generated by the dynamical system \ref{['DynSys']}
  8. Conclusion, perspectives
  9. Statements and Declarations
  10. Appendix

Key Result

Theorem 1

Suppose that the regularization parameter has the form $\epsilon(t)=\frac{a}{t^p},\,a,p>0$ and for some starting points $u_0, \, v_0\in\mathcal{H}$ let $x : [t_0, +\infty[ \to \mathcal{H}$ be the unique global solution of DynSys. Then, the following results hold.

Theorems & Definitions (23)

  • Theorem 1
  • Theorem 2
  • Proof 1
  • Theorem 3
  • Proof 2
  • Remark 4
  • Theorem 5
  • Proof 3
  • Theorem 6
  • Proof 4
  • ...and 13 more