Computer-Assisted Search for Differential Equations Corresponding to Optimization Methods and Their Convergence Rates
Atsushi Tabei, Ken'ichiro Tanaka
TL;DR
This work tackles the challenge of systematically deriving Lyapunov functions and convergence rates for continuous-time dynamical systems that model optimization methods. It introduces a computer-assisted, symbolic-framework that exhaustively explores Lyapunov-constructing operations by representing p and q as PSD matrices P and Q and combining a finite set of algebraic updates. Applying the method to six classes of ODEs corresponding to optimization schemes (including Hessian-involved dynamics and generalized NAG variants) reproduces known rates and uncovers new convergence rates, while highlighting that Hessian terms do not universally improve rates within the tested classes. The paper also presents a restart scheme that achieves linear convergence under μ-strong convexity, illustrating practical gains from the automated framework and motivating future work on discretization and broader performance measures.
Abstract
Let $f:\mathbb{R}^n \to \mathbb{R}$ be a continuously differentiable convex function with its minimizer denoted by $x_*$ and optimal value $f_* = f(x_*)$. Optimization algorithms such as the gradient descent method can often be interpreted in the continuous-time limit as differential equations known as continuous dynamical systems. Analyzing the convergence rate of $f(x) - f_*$ in such systems often relies on constructing appropriate Lyapunov functions. However, these Lyapunov functions have been designed through heuristic reasoning rather than a systematic framework. Several studies have addressed this issue. In particular, Suh, Roh, and Ryu (2022) proposed a constructive approach that involves introducing dilated coordinates and applying integration by parts. Although this method significantly improves the process of designing Lyapunov functions, it still involves arbitrary choices among many possible options, and thus retains a heuristic nature in identifying Lyapunov functions that yield the best convergence rates. In this study, we propose a systematic framework for exploring these choices computationally. More precisely, we propose a brute-force approach using symbolic computation by computer algebra systems to explore every possibility. By formulating the design of Lyapunov functions for continuous dynamical systems as an optimization problem, we aim to optimize the Lyapunov function itself. As a result, our framework successfully reproduces many previously reported results and, in several cases, discovers new convergence rates that have not been shown in the existing studies.