e-COP : Episodic Constrained Optimization of Policies
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Sahil Singla
TL;DR
The paper advances constrained RL in episodic, finite-horizon MDPs by introducing e-COP, a PPO-like policy optimization method anchored to a novel episodic policy-difference lemma. It replaces Hessian-based updates with a quadratic damping penalty and a ReLU-like constraint penalty, yielding a tractable, clipped surrogate that maintains equivalence to the CMDP optimum under appropriate scaling. The approach provides both theoretical guarantees and strong empirical performance on Safety Gym and MuJoCo benchmarks, outperforming or matching state-of-the-art baselines adapted to episodic settings. The proposed method is easy to integrate into existing RL pipelines and holds promise for safety-critical applications including RL from human feedback for large models.
Abstract
In this paper, we present the $\texttt{e-COP}$ algorithm, the first policy optimization algorithm for constrained Reinforcement Learning (RL) in episodic (finite horizon) settings. Such formulations are applicable when there are separate sets of optimization criteria and constraints on a system's behavior. We approach this problem by first establishing a policy difference lemma for the episodic setting, which provides the theoretical foundation for the algorithm. Then, we propose to combine a set of established and novel solution ideas to yield the $\texttt{e-COP}$ algorithm that is easy to implement and numerically stable, and provide a theoretical guarantee on optimality under certain scaling assumptions. Through extensive empirical analysis using benchmarks in the Safety Gym suite, we show that our algorithm has similar or better performance than SoTA (non-episodic) algorithms adapted for the episodic setting. The scalability of the algorithm opens the door to its application in safety-constrained Reinforcement Learning from Human Feedback for Large Language or Diffusion Models.