Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frictionless Hamiltonian Descent and Coordinate Hamiltonian Descent for Strongly Convex Quadratic Problems and Beyond

Jun-Kun Wang

TL;DR

The paper addresses optimization by importing frictionless Hamiltonian dynamics as a direct optimization analogue of Hamiltonian Monte Carlo, specifically targeting strongly convex quadratic problems and beyond. It develops Frictionless-HD, which uses energy-conserving Hamiltonian flow with periodic resets to guarantee descent, and shows a matrix-power–vector update structure that yields accelerated convergence on quadratics. The work then unifies and extends classical linear-solvers within Frictionless-Coordinate CHD and its parallel variant, showing Gauss-Seidel, SOR, Jacobi, and weighted Jacobi emerge from particular integration times, along with a criterion guaranteeing convergence. It further demonstrates potential applicability to certain nonconvex objectives under gradient-dominance conditions, and discusses limitations and avenues for future research, including numerical integration and broader nonconvex settings. Overall, the framework provides a novel, mechanics-inspired perspective and convergence guarantees for both traditional and parallel coordinate methods, with evidence of accelerated behavior on key problem classes.

Abstract

We propose an optimization algorithm called Frictionless Hamiltonian Descent, which is a direct counterpart of classical Hamiltonian Monte Carlo in sampling. We find that Frictionless Hamiltonian Descent for solving strongly convex quadratic problems exhibits a novel update scheme that involves matrix-power-vector products. We also propose Frictionless Coordinate Hamiltonian Descent and its parallelizable variant, which turns out to encapsulate the classical Gauss-Seidel method, Successive Over-relaxation, Jacobi method, and more, for solving a linear system of equations. The result not only offers a new perspective on these existing algorithms but also leads to a broader class of update schemes that guarantee the convergence. Finally, we also highlight the potential of Frictionless Hamiltonian Descent beyond quadratics by studying solving certain non-convex functions, where Frictionless Hamiltonian Descent can find a global optimal point.

Frictionless Hamiltonian Descent and Coordinate Hamiltonian Descent for Strongly Convex Quadratic Problems and Beyond

TL;DR

The paper addresses optimization by importing frictionless Hamiltonian dynamics as a direct optimization analogue of Hamiltonian Monte Carlo, specifically targeting strongly convex quadratic problems and beyond. It develops Frictionless-HD, which uses energy-conserving Hamiltonian flow with periodic resets to guarantee descent, and shows a matrix-power–vector update structure that yields accelerated convergence on quadratics. The work then unifies and extends classical linear-solvers within Frictionless-Coordinate CHD and its parallel variant, showing Gauss-Seidel, SOR, Jacobi, and weighted Jacobi emerge from particular integration times, along with a criterion guaranteeing convergence. It further demonstrates potential applicability to certain nonconvex objectives under gradient-dominance conditions, and discusses limitations and avenues for future research, including numerical integration and broader nonconvex settings. Overall, the framework provides a novel, mechanics-inspired perspective and convergence guarantees for both traditional and parallel coordinate methods, with evidence of accelerated behavior on key problem classes.

Abstract

We propose an optimization algorithm called Frictionless Hamiltonian Descent, which is a direct counterpart of classical Hamiltonian Monte Carlo in sampling. We find that Frictionless Hamiltonian Descent for solving strongly convex quadratic problems exhibits a novel update scheme that involves matrix-power-vector products. We also propose Frictionless Coordinate Hamiltonian Descent and its parallelizable variant, which turns out to encapsulate the classical Gauss-Seidel method, Successive Over-relaxation, Jacobi method, and more, for solving a linear system of equations. The result not only offers a new perspective on these existing algorithms but also leads to a broader class of update schemes that guarantee the convergence. Finally, we also highlight the potential of Frictionless Hamiltonian Descent beyond quadratics by studying solving certain non-convex functions, where Frictionless Hamiltonian Descent can find a global optimal point.
Paper Structure (18 sections, 24 theorems, 104 equations, 2 figures, 6 algorithms)

This paper contains 18 sections, 24 theorems, 104 equations, 2 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Theoretical analysis of Frictionless Hamiltonian Descent for strongly convex quadratic problems
  3. Frictionless Coordinate Hamiltonian Descent
  4. Parallel Frictionless Coordinate Hamiltonian Descent
  5. On minimizing non-convex functions
  6. Limitations and outlooks
  7. Proof of lemmas and theorems in Section \ref{['sec:Hami']}
  8. Proof of Lemma \ref{['lem:per_update']}
  9. Chebyshev Polynomial
  10. Proof of Theorem \ref{['thm:HD']}
  11. Proof of lemmas and theorems in Section \ref{['sec:Co-Hami']}
  12. Proof of Lemma \ref{['lem:cord_dym']}
  13. Proof of Lemma \ref{['lem:cod']}
  14. Proof of Theorem \ref{['thm:converges']}
  15. Randomized Frictionless Coordinate Hamiltonian Descent
  16. ...and 3 more sections

Key Result

Lemma 1.1

The time derivative of the Hamiltonian satisfies $\frac{dH}{dt} = 0$ along the Hamiltonian Flow (flow).

Figures (2)

  • Figure 1: HD vs. GF vs. NAF
  • Figure 2: $\psi(x) = \frac{ \cos \left( \frac{\pi}{2} \sqrt{x} \right) }{1 - x}$ v.s. $x$. (a) $\psi(x)$ v.s. $x$ where $x \in [0,1]$, (b) $\psi(x)$ v.s. $x$ where $x \in [0,100]$.

Theorems & Definitions (49)

  • Lemma 1.1
  • proof
  • Lemma 1.2
  • proof
  • Lemma 2.1
  • Lemma 2.2
  • Theorem 2.3
  • Theorem 2.4
  • proof
  • Lemma 3.1
  • ...and 39 more