Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations

TL;DR

This work formulates AdaGrad, RMSProp, and Adam as first-order integro-differential equations in continuous time, encoding memory via nonlocal kernels. The IDEs reproduce the dynamics of the discrete optimizers, enabling rigorous stability and convergence analysis with Lyapunov and LaSalle tools; convex objectives exhibit exponential convergence while nonconvex cases admit PL/KL-type rates depending on the memory and smoothness. Theoretical results are complemented by numerical simulations using an IDESolver in JAX, which demonstrate strong agreement with the discrete algorithms across both convex and nonconvex settings and reveal how memory strength shapes convergence rates. Overall, the integro-differential perspective provides a principled bridge between discrete adaptive methods and continuous dynamical systems, offering insights for memory-driven optimization and potential nonlocal extensions in learning dynamics.

Abstract

In this paper, we propose a continuous-time formulation for the AdaGrad, RMSProp, and Adam optimization algorithms by modeling them as first-order integro-differential equations. We perform numerical simulations of these equations, along with stability and convergence analyses, to demonstrate their validity as accurate approximations of the original algorithms. Our results indicate a strong agreement between the behavior of the continuous-time models and the discrete implementations, thus providing a new perspective on the theoretical understanding of adaptive optimization methods.

Figures (15)

• Figure 1: Convergence of $\theta(t)$ using the first-order nonlocal continuous AdaGrad method. The plot illustrates the convergence trajectories for minimizing the function $(\theta - 4)^2$ using two different learning rates: $0.1$ (left) and $0.01$ (right). At a higher learning rate ($0.1$), the algorithm rapidly converges to the target value $\theta = 4$, stabilizing in under 1,500 k-iterations. With a lower learning rate ($0.01$), the convergence is more gradual, reaching the target around 100,000 k-iterations.
• Figure 2: Accumulated gradients $G(t)$ convergence trajectories using the first-order nonlocal continuous AdaGrad method. The plot shows that the nonlocal continuous AdaGrad method exhibits a nearly identical gradient accumulation behavior to the conventional AdaGrad method, with rapid convergence and stabilization at both learning rates.
• Figure 3: Convergence of $\theta(t)$ using the first-order nonlocal continuous RMSProp method. The plot shows the convergence to the minimum value of $\theta = 4$ for the convex function $(\theta - 4)^2$. The left subplot corresponds to a learning rate of 0.1, while the right subplot uses a learning rate of 0.01. With a learning rate of 0.1, $\beta = 0.0$ exhibits more noticeable oscillations, whereas $\beta = 0.9$ and $0.99$ converge more smoothly. For a learning rate of 0.01, a slight destabilization occurs for $\beta = 0.9$ as the solution approaches the final result, caused by the numerical method. At higher learning rates, slight differences can be observed between the models: for $\beta = 0.0$, the oscillations in the discrete case start immediately upon reaching the minimum, whereas in the continuous case, they take a few k-iterations to begin. On the other hand, for $\beta = 0.99$, the curve is less pronounced in the discrete model compared to the continuous one.
• Figure 4: Convergence trajectories of $v(t)$ for the first-order nonlocal continuous RMSProp method. This plot illustrates the convergence of the squared gradient moving average $v(t)$. The left subplot represents a learning rate of 0.1, while the right subplot corresponds to a learning rate of 0.01. For $\beta$ values of 0.0 and 0.9, a slight initial bump is noticeable, but both eventually decay towards zero. The main differences between the continuous and discrete models appear at a learning rate of $0.1$, where the values are slightly higher for $\beta = 0.9$ and $\beta = 0.99$, and for $\beta = 0.0$ a small bump is observed instead of a direct descent.
• Figure 5: Convergence of $\theta(t)$ using the first-order nonlocal continuous Adam method. The plot illustrates the convergence trajectories of $\theta$-values for the first-order nonlocal continuous Adam model under different parameter settings of $\beta_1$ and $\beta_2$ with two distinct learning rates ($0.1$ and $0.01$). For $\beta_1 = 0.9$ and a learning rate of 0.1, a noticeable oscillation is observed, requiring a longer time to stabilize at the minimum value. This behavior improves when the learning rate is reduced to 0.01.

Theorems & Definitions (46)

• Proposition 1: Nonlocal Continuous Dynamics of AdaGrad
• proof
• Proposition 2: Nonlocal Continuous Dynamics of RMSProp
• proof
• Proposition 3: Nonlocal Continuous Dynamics of Adam
• proof
• Definition 1
• Lemma 1
• proof
• Lemma 2: Bounds on $K_\nu$