Trajectory-Oriented Policy Optimization with Sparse Rewards
Guojian Wang, Faguo Wu, Xiao Zhang
TL;DR
The paper addresses the challenge of learning under sparse rewards in deep reinforcement learning by leveraging offline demonstration trajectories as soft guidance. It introduces Trajectory Oriented Policy Optimization (TOPO), which uses a maximum mean discrepancy (MMD) based trajectory distance to align the agent's state-action visitation distribution with the demonstrations and reformulates policy optimization as a distance-constrained objective. This constrained problem is converted into a policy-gradient algorithm with an intrinsic distance reward $r^{(i)}(s,a)$, enabling efficient exploration without dense reward design or perfect demonstrations. Empirical results on discrete tasks like Key-Door-Treasure and continuous tasks such as SparseHalfCheetah and SparseHopper show TOPO outperforms baseline methods in exploration speed and final policy quality, highlighting its practical impact for sparse reward RL in both discrete and continuous domains.
Abstract
Mastering deep reinforcement learning (DRL) proves challenging in tasks featuring scant rewards. These limited rewards merely signify whether the task is partially or entirely accomplished, necessitating various exploration actions before the agent garners meaningful feedback. Consequently, the majority of existing DRL exploration algorithms struggle to acquire practical policies within a reasonable timeframe. To address this challenge, we introduce an approach leveraging offline demonstration trajectories for swifter and more efficient online RL in environments with sparse rewards. Our pivotal insight involves treating offline demonstration trajectories as guidance, rather than mere imitation, allowing our method to learn a policy whose distribution of state-action visitation marginally matches that of offline demonstrations. We specifically introduce a novel trajectory distance relying on maximum mean discrepancy (MMD) and cast policy optimization as a distance-constrained optimization problem. We then illustrate that this optimization problem can be streamlined into a policy-gradient algorithm, integrating rewards shaped by insights from offline demonstrations. The proposed algorithm undergoes evaluation across extensive discrete and continuous control tasks with sparse and misleading rewards. The experimental findings demonstrate the significant superiority of our proposed algorithm over baseline methods concerning diverse exploration and the acquisition of an optimal policy.