Guided Policy Optimization under Partial Observability
Yueheng Li, Guangming Xie, Zongqing Lu
TL;DR
The paper tackles reinforcement learning in partially observable environments ($POMDP$) where training-time privileged information can be leveraged but is hard to utilize. It introduces Guided Policy Optimization (GPO), a co-training framework that learns a guider with access to privileged data and a learner that imitates the guider, with backtracking to ensure feasible imitation and provable learner optimality comparable to direct RL. Two practical variants, GPO-Penalty and GPO-Clip, are proposed, incorporating KL-based penalties or double clipping and a backtracking mask, alongside a theoretical link to standard RL updates for the learner. Empirically, GPO outperforms baselines across didactic tasks, Brax continuous control under noise/partial observability, and POPGym memory tasks, demonstrating robust performance and effective use of privileged information with memory demands; future work could extend the framework to multi-agent settings and refine guidance schedules.
Abstract
Reinforcement Learning (RL) in partially observable environments poses significant challenges due to the complexity of learning under uncertainty. While additional information, such as that available in simulations, can enhance training, effectively leveraging it remains an open problem. To address this, we introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner. The guider takes advantage of privileged information while ensuring alignment with the learner's policy that is primarily trained via imitation learning. We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches. Empirical evaluations show strong performance of GPO across various tasks, including continuous control with partial observability and noise, and memory-based challenges, significantly outperforming existing methods.