Conformal Symplectic Optimization for Stable Reinforcement Learning

TL;DR

This work casts neural network training as the evolution of a conformal Hamiltonian system and introduces RAD, a Relativistic Adaptive Gradient Descent algorithm that enforces a finite update speed via a relativistic kinetic energy and per-parameter dynamics. By using conformal symplectic discretization and a multi-particle interpretation, RAD achieves improved long-term stability and faster convergence in reinforcement learning, with convergence guarantees under general nonconvex stochastic settings. RAD naturally links to ADAM when the speed coefficient is fixed, offering a physics-based interpretation of adaptive gradient methods and a principled transition from ADAM-like behavior to conformal symplectic stability. Empirical results across MuJoCo, Atari, and autonomous driving tasks show RAD outperforming multiple baselines, including ADAM, with substantial TAR improvements and robust performance under gradient variance and observation noise.

Abstract

Training deep reinforcement learning (RL) agents necessitates overcoming the highly unstable nonconvex stochastic optimization inherent in the trial-and-error mechanism. To tackle this challenge, we propose a physics-inspired optimization algorithm called relativistic adaptive gradient descent (RAD), which enhances long-term training stability. By conceptualizing neural network (NN) training as the evolution of a conformal Hamiltonian system, we present a universal framework for transferring long-term stability from conformal symplectic integrators to iterative NN updating rules, where the choice of kinetic energy governs the dynamical properties of resulting optimization algorithms. By utilizing relativistic kinetic energy, RAD incorporates principles from special relativity and limits parameter updates below a finite speed, effectively mitigating abnormal gradient influences. Additionally, RAD models NN optimization as the evolution of a multi-particle system where each trainable parameter acts as an independent particle with an individual adaptive learning rate. We prove RAD's sublinear convergence under general nonconvex settings, where smaller gradient variance and larger batch sizes contribute to tighter convergence. Notably, RAD degrades to the well-known adaptive moment estimation (ADAM) algorithm when its speed coefficient is chosen as one and symplectic factor as a small positive value. Experimental results show RAD outperforming nine baseline optimizers with five RL algorithms across twelve environments, including standard benchmarks and challenging scenarios. Notably, RAD achieves up to a 155.1% performance improvement over ADAM in Atari games, showcasing its efficacy in stabilizing and accelerating RL training.

Figures (8)

• Figure 1: Learning curves of CartPole-v1, wherein the TAR indicates the policy performance and the smoothness of Hamiltonian descending implies the training stability. The solid lines correspond to the mean, the shaded regions correspond to the 95% confidence interval over five runs, and $\Delta H=H-\min H$.
• Figure 2: Policy performance of SAC on MuJoCo tasks. The solid lines correspond to the mean, and the shaded regions correspond to the 95% confidence interval over five runs.
• Figure 3: Policy performance on Atari games. The solid lines correspond to the mean, and the shaded regions correspond to the 95% confidence interval over five runs.
• Figure 4: Performance comparison among SOTA optimizers. The solid lines correspond to the mean, the shaded regions correspond to the 95% confidence interval over five runs.
• Figure 5: Policy performance on Walker2d-v3 under observation noise. For each noise level, the observation noise is randomly generated from standard Gaussian distributions with standard deviations of 0, 0.001, 0.005, and 0.01, respectively.

Theorems & Definitions (13)

• Remark 1
• Corollary 1
• Corollary 2
• Corollary 3
• Lemma 1
• Theorem 1
• Corollary 4: Convergence of RAD
• Corollary 5
• Corollary 6: Convergence of ADAM
• Remark 2