Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward Constrained Policy Optimization

Chen Tessler, Daniel J. Mankowitz, Shie Mannor

TL;DR

RCPO tackles the challenge of enforcing general constraints in reinforcement learning by embedding a penalty signal into a multi-timescale learning framework. It blends a discounted penalty-guided critic with a slower-gradient penalty updater, yielding convergence to constraint-satisfying policies under mild assumptions. The approach is validated on grid-world and six Mujoco robotics tasks, showing improved stability, sample efficiency, and feasibility compared to reward shaping and Lagrangian-dual baselines. The work highlights the benefits of adaptive penalty learning for safe, high-performance RL in complex domains.

Abstract

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Reward Constrained Policy Optimization

TL;DR

RCPO tackles the challenge of enforcing general constraints in reinforcement learning by embedding a penalty signal into a multi-timescale learning framework. It blends a discounted penalty-guided critic with a slower-gradient penalty updater, yielding convergence to constraint-satisfying policies under mild assumptions. The approach is validated on grid-world and six Mujoco robotics tasks, showing improved stability, sample efficiency, and feasibility compared to reward shaping and Lagrangian-dual baselines. The work highlights the benefits of adaptive penalty learning for safe, high-performance RL in complex domains.

Abstract

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Paper Structure

This paper contains 29 sections, 3 theorems, 19 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Process (MDP)
  4. Constrained MDPs
  5. Parametrized Policies
  6. Constrained Policy Optimization
  7. Estimating the gradient
  8. Reward Constrained Policy Optimization
  9. Actor Critic Requirements
  10. Penalized reward functions
  11. Experiments
  12. Mars Rover
  13. Domain Description
  14. Experiment Description
  15. Experiment Analysis
  16. ...and 14 more sections

Key Result

Theorem 1

Under Assumption step_sizes, as well as the standard stability assumption for the iterates and bounded noise borkar2008stochastic, the iterates $(\theta_n, \lambda_n)$ converge to a fixed point (a local minima) almost surely.

Figures (2)

  • Figure 1: Mars Rover domain and policy illustration. As $\alpha$ decreases, the agent is required to learn a safer policy.
  • Figure 3: Mujoco with torque constraints. The dashed line represents the maximal allowed value. Results are considered valid only if they are at or below the threshold. RCPO is our approach, whereas each $\lambda$ value is a PPO simulation with a fixed penalty coefficient. Y axis is the average reward and the X axis represents the number of samples (steps).

Theorems & Definitions (6)

  • Definition 1
  • Theorem 1
  • Lemma 1
  • Definition 2
  • Definition 3
  • Theorem 2