Proximal Policy Optimization with Evolutionary Mutations

TL;DR

POEM tackles premature convergence in Proximal Policy Optimization by introducing an adaptive exploration mechanism that monitors the KL divergence between the current policy and a moving-average reference. When stagnation is detected, POEM injects adaptive evolutionary mutations into the policy parameters to promote exploration, then resumes PPO updates, preserving stability. Across four OpenAI Gym tasks, POEM significantly outperforms PPO on three environments, demonstrating the value of KL-based diversity triggers for balancing exploration and exploitation in policy gradient methods. This approach suggests a practical route to enhance sample efficiency and robustness in reinforcement learning applications where reward structures vary in density and structure.

Abstract

Proximal Policy Optimization (PPO) is a widely used reinforcement learning algorithm known for its stability and sample efficiency, but it often suffers from premature convergence due to limited exploration. In this paper, we propose POEM (Proximal Policy Optimization with Evolutionary Mutations), a novel modification to PPO that introduces an adaptive exploration mechanism inspired by evolutionary algorithms. POEM enhances policy diversity by monitoring the Kullback-Leibler (KL) divergence between the current policy and a moving average of previous policies. When policy changes become minimal, indicating stagnation, POEM triggers an adaptive mutation of policy parameters to promote exploration. We evaluate POEM on four OpenAI Gym environments: CarRacing, MountainCar, BipedalWalker, and LunarLander. Through extensive fine-tuning using Bayesian optimization techniques and statistical testing using Welch's t-test, we find that POEM significantly outperforms PPO on three of the four tasks (BipedalWalker: t=-2.0642, p=0.0495; CarRacing: t=-6.3987, p=0.0002; MountainCar: t=-6.2431, p<0.0001), while performance on LunarLander is not statistically significant (t=-1.8707, p=0.0778). Our results highlight the potential of integrating evolutionary principles into policy gradient methods to overcome exploration-exploitation tradeoffs.

Figures (5)

• Figure 1: Cumulative reward per step on CarRacing-v3 for POEM and PPO across 15 evaluation episodes. (a) POEM: shows low variance across seeds, reliably progresses through the track, and completes all but four runs. (b) PPO: frequently becomes stuck on turns and fails to complete any episode.
• Figure 2: Cumulative reward per step on MountainCarContinuous-v0 for POEM and PPO across 15 evaluation episodes . (a) POEM: successfully explores the environment, escapes the valley, and consistently reaches the goal incurring minimal negative reward. (b) PPO: fails to solve the task in all episodes, reaching the maximum timestep limit or exiting the bounds of the course and incurring the full 1000-step cumulative cost. Note Y scales are different between graphs and final reward of +100 for completing the task is not graphed for readability.
• Figure 3: Cumulative reward per step for POEM and PPO on BipedalWalker-v3 across 15 evaluation episodes. (a) POEM: reward trajectories show low variance across seeds, indicating stable performance. POEM reliably progresses across the terrain, with all but three episodes completing the course, and all successful episodes finishing in under 1200 steps. (b) PPO: completes the course in only 7 out of 15 episodes, with the remainder resulting in failure. No PPO run completes the course in fewer than 1200 steps.
• Figure 4: Cumulative reward per step for POEM and PPO on LunarLander-v3 across 15 evaluation episodes. (a) POEM: reward increases rapidly during the first 100 steps as the agent stabilizes its descent, followed by a plateau where the lander fine-tunes its position before performing a controlled landing. (b) PPO: exhibits a more gradual reward increase, reflecting a strategy that involves searching for a landing region at higher altitude before descending and attempting the final landing maneuver.
• Figure 5: Episode-wise total rewards for PPO (blue) and POEM (orange) across 15 runs. POEM wins the vast majority of runs; PPO only exceeds POEM in 1 episode of BipedalWalker and 4 episodes of LunarLander.