An Approximate Ascent Approach To Prove Convergence of PPO

TL;DR

This work provides a theory-backed reinterpretation of PPO as biased policy-gradient ascent with surrogate gradients under random reshuffling, enabling convergence analysis for cycle-based, multi-epoch updates that reuse on-policy data. It introduces a surrogate gradient $g_ ext{PPO}^ ext{clip}$ and proves a bias bound $| abla J(\theta) - g_ ext{PPO}^ ext{clip}(\theta,\theta_{\text{old}})| \le R\,|\theta-\theta_{\text{old}}|$, with convergence guarantees in both deterministic and stochastic RR settings. The paper also identifies tail-mass collapse in truncated GAE and proposes finite-time and termination-time GAEs to renormalize weights, showing termination-time GAE improves learning in Lunar Lander. Empirical results corroborate faster, more stable learning under terminal-signal environments, and the analysis paves the way for further theoretical and methodological refinements of PPO-like algorithms.

Abstract

Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling to prove a convergence theorem for PPO that sheds light on PPO's success. Additionally, we identify a previously overlooked issue in truncated Generalized Advantage Estimation commonly used in PPO. The geometric weighting scheme induces infinite mass collapse onto the longest $k$-step advantage estimator at episode boundaries. Empirical evaluations show that a simple weight correction can yield substantial improvements in environments with strong terminal signal, such as Lunar Lander.

Figures (8)

• Figure 1: Schematic view of PPO vs. A2C policy parameter updates. Blue arrows are policy gradient $\nabla_\theta J(\theta)$ steps, orange arrows cycles of increasingly biased surrogate gradient $g_\text{PPO}^\text{clip}(\theta,\theta_\text{old})$ steps. For stochastic approximations blue dots are resampling times.
• Figure 2: GAE tail-mass collapse. Comparison of the weights assigned to the $k$-step advantage estimators $\hat{\mathbb A}_t^{(k)}$ by truncated GAE and our finite-time (renormalized) estimator. Each panel corresponds to a different remaining horizon $\tau-t$ and shows how, for truncated GAE, the geometric tail mass collapses onto the last available estimator $\hat{\mathbb A}_t^{(\tau-t-1)}$, while the finite-time variant redistributes this mass over the observable $k$-step estimators via finite-horizon renormalization .
• Figure 3: LunarLander-v3 learning curves with default PPO hyperparameters from Stable-Baselines3 Zoo. Left (middle): mean evaluation (discounted) return, i.e. sum of (discounted) rewards per episode. Right: mean evaluation episode length. Curves are averaged over 20 seeds with standard errors of seeds as the shaded regions. Both methods use identical default hyperparameters; only the advantage estimator differs (truncated GAE vs. our finite-time GAEs). Additional metrics and ablations are reported in Appendix \ref{['sec: finite-time-gae']}
• Figure 4: Covariance heatmaps differences for truncated GAE (PPO) vs. finite-time GAE (us) at fixed $\gamma = 0.999$, $\lambda=0.95$ and $\tau=200$ (left) and $\tau=1000$ (right) under simplified assumptions (see Appendix \ref{['sec: finite-time-gae']}). Color scale shows $\operatorname{Cov}[\hat{\mathbb A}_t^\tau, \hat{\mathbb A}_s^\tau]-\operatorname{Cov}[\hat{\mathbb A}_t, \hat{\mathbb A}_s]$.
• Figure 5: LunarLander-v3 evaluation learning curves under three hyperparameter (HP) regimes. Columns report evaluation return (sum of rewards), discounted evaluation return (sum of discounted rewards), and evaluation episode length. Curves are averaged over 20 seeds with standard errors of seeds as the shaded region. Rows correspond to hyperparameter regimes: Top: default PPO hyperparameters from the Stable-Baselines3 Zoo. Middle: best hyperparameters found separately for each method via hyperparameter optimization (100 trials, 3 seeds per trial). Bottom: all methods evaluated using the hyperparameters obtained from the truncated-GAE hyperparameter optimization (same hyperparameters for all methods).

Theorems & Definitions (84)

• Remark 2.1
• Remark 4.1
• Theorem 4.2: Surrogate gradient bias control
• proof : Sketch of proof
• Theorem 5.1: Deterministic PPO convergence
• Remark 6.1
• Theorem 6.2
• Proposition 7.1: Tail-mass collapse of truncated GAE
• Definition 7.2: Finite-time GAEs
• Proposition 7.3