Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
Qingkai Liang, Fanyu Que, Eytan Modiano
TL;DR
The paper addresses sample inefficiency in constrained reinforcement learning by introducing Accelerated Primal-Dual Optimization (APDO), which adds a one-time off-policy dual adjustment to the standard primal-dual framework for CMDPs. By training a near-optimal dual variable λOFF from replay-buffer data and applying it after K_adj iterations, APDO accelerates dual convergence while preserving on-policy primal updates. Empirical results on a safety-constrained MuJoCo task show APDO achieves faster convergence and better sample efficiency than state-of-the-art methods like PDO and CPO, with robust constraint enforcement. The work also discusses trade-offs in choosing K_adj and outlines directions for theoretical analysis and safety-aware extensions.
Abstract
Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updates parameters in primal and dual spaces in turn. Existing methods for CMDPs only use on-policy data for dual updates, which results in sample inefficiency and slow convergence. In this paper, we propose a policy search method for CMDPs called Accelerated Primal-Dual Optimization (APDO), which incorporates an off-policy trained dual variable in the dual update procedure while updating the policy in primal space with on-policy likelihood ratio gradient. Experimental results on a simulated robot locomotion task show that APDO achieves better sample efficiency and faster convergence than state-of-the-art approaches for CMDPs.