Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

TL;DR

This work targets reinforcement learning in environments with delayed rewards by marrying an offline, demonstration-based policy with online PPO through a mixing framework that guarantees monotonic improvement and a bounded performance gap $\frac{2\varsigma\gamma\alpha^2}{(1-\gamma)^2}$. It further introduces a reward-shaping mechanism based on Time Window Temporal Logic (TWTL) that provides immediate, structured feedback via a predictor-driven robustness score while preserving the original optimal policy through potential-based shaping. The approach combines an LSTM-based task predictor with a mixed-policy objective, and provides formal convergence guarantees alongside an APPO algorithm, demonstrating improved learning speed and final performance on Lunar Lander and Inverted Pendulum, even when offline policies are suboptimal. Overall, the method offers a principled way to leverage expert data and temporal objectives to accelerate learning in tasks with long-horizon consequences and sparse credit assignment, with potential applicability to complex temporal tasks and adaptive mixing strategies.

Abstract

In this paper, we tackle the challenging problem of delayed rewards in reinforcement learning (RL). While Proximal Policy Optimization (PPO) has emerged as a leading Policy Gradient method, its performance can degrade under delayed rewards. We introduce two key enhancements to PPO: a hybrid policy architecture that combines an offline policy (trained on expert demonstrations) with an online PPO policy, and a reward shaping mechanism using Time Window Temporal Logic (TWTL). The hybrid architecture leverages offline data throughout training while maintaining PPO's theoretical guarantees. Building on the monotonic improvement framework of Trust Region Policy Optimization (TRPO), we prove that our approach ensures improvement over both the offline policy and previous iterations, with a bounded performance gap of $(2ςγα^2)/(1-γ)^2$, where $α$ is the mixing parameter, $γ$ is the discount factor, and $ς$ bounds the expected advantage. Additionally, we prove that our TWTL-based reward shaping preserves the optimal policy of the original problem. TWTL enables formal translation of temporal objectives into immediate feedback signals that guide learning. We demonstrate the effectiveness of our approach through extensive experiments on an inverted pendulum and a lunar lander environments, showing improvements in both learning speed and final performance compared to standard PPO and offline-only approaches.

Figures (2)

• Figure 1: The Actor-Critic RL framework. The Actor's architecture, policy $\pi_\theta$, consists of an offline policy, $\pi_\rho$, and an adaptive policy, $\pi_\beta$, where the two policies are mixed using the parameters of the FCL, $\alpha$. The critic consists of an MLP that approximates the value function. The task predictor LSTM network and the reward shaping are depicted in cyan.
• Figure 2: Training performance comparison across different variants of PPO in LunarLander-v2 and Pendulum environments. For LunarLander-v2 (left $y$-axis), we compare vanilla PPO against variants with reward shaping and mixing. The results show that combining reward shaping with policy mixing achieves faster learning and better performance compared to baseline PPO and reward shaping alone. For Pendulum (right $y$-axis, dashed lines), we include PPO with and without mixing for reference. Training steps are shown in millions on the $x$-axis.

Theorems & Definitions (12)

• definition 1
• definition 2
• definition 3: Concrete Time Reward
• lemma 1
• proof
• lemma 2
• proof
• lemma 3: abbele2015TRPO
• proposition 1
• proof