AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization
Soham Sane
TL;DR
PPO's effectiveness hinges on accurate advantage estimates, but raw advantages often exhibit high variance and scaling issues. AM-PPO introduces an adaptive advantage modulation mechanism that uses an EMA-driven alpha controller and a bounded tanh gate to reshape $A^{\text{raw}}$ into $A_{\text{mod}}$, which then guides both policy and value learning. The approach yields more stable learning trajectories and improved final rewards on continuous-control benchmarks, while reducing the need for aggressive optimizer clipping. The framework suggests a generalizable strategy for conditioning learning signals in RL, with potential extensions to AGRAM, value-function-free settings, and off-policy algorithms.
Abstract
Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm that heavily relies on accurate advantage estimates for stable and efficient training. However, raw advantage signals can exhibit significant variance, noise, and scale-related issues, impeding optimal learning performance. To address this challenge, we introduce Advantage Modulation PPO (AM-PPO), a novel enhancement of PPO that adaptively modulates advantage estimates using a dynamic, non-linear scaling mechanism. This adaptive modulation employs an alpha controller that dynamically adjusts the scaling factor based on evolving statistical properties of the advantage signals, such as their norm, variance, and a predefined target saturation level. By incorporating a tanh-based gating function driven by these adaptively scaled advantages, AM-PPO reshapes the advantage signals to stabilize gradient updates and improve the conditioning of the policy gradient landscape. Crucially, this modulation also influences value function training by providing consistent and adaptively conditioned learning targets. Empirical evaluations across standard continuous control benchmarks demonstrate that AM-PPO achieves superior reward trajectories, exhibits sustained learning progression, and significantly reduces the clipping required by adaptive optimizers. These findings underscore the potential of advantage modulation as a broadly applicable technique for enhancing reinforcement learning optimization.