Moduli space of optimization algorithms
Dmitry Pasechnyuk-Vilensky, Martin Takáč
TL;DR
The paper introduces a moduli-space view of optimization algorithms, treating updates as discrete connections whose drift and diffusion channels encode reversibility and energy dissipation. A flat (jet-flat) curvature regime yields minimal dissipation and optimal stability, while gauge corrections enable higher-order, energy-monotone variants; the framework unifies gradient, proximal, and momentum methods and extends to stochastic/adaptive settings. A variational reformulation via a Yang–Mills–type curvature and an isomonodromic (tau-function) perspective provides exact nonasymptotic bounds and links acceleration to extremal filter theory (Chebyshev/Zolotarev). The results include a minimax principle for discrete holonomy, a semi-algebraic moduli model, and adaptive geometry for curvature-aware preconditioning, with concrete algorithms and gauge-solve strategies (Sylvester equation) for practical calibration. The isomonodromic theory further connects discrete update dynamics to integrable systems and Riemann–Hilbert methods, yielding a deep, nonasymptotic, algebraic understanding of stability and acceleration in optimization.
Abstract
We introduce a geometric and operator-theoretic formalism viewing optimization algorithms as discrete connections on a space of update operators. Each iterative method is encoded by two coupled channels-drift and diffusion-whose algebraic curvature measures the deviation from ideal reversibility and determines the attainable order of accuracy. Flat connections correspond to methods whose updates commute up to higher order and thus achieve minimal numerical dissipation while preserving stability. The formalism recovers classical gradient, proximal, and momentum schemes as first-order flat cases and extends naturally to stochastic, preconditioned, and adaptive algorithms through perturbations controlled by curvature order. Explicit gauge corrections yield higher-order variants with guaranteed energy monotonicity and noise stability. The associated determinantal and isomonodromic formulations yield exact nonasymptotic bounds and describe acceleration effects via Painlevé-type invariants and Stokes corrections.