Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

Daniel J. Mankowitz, Dan A. Calian, Rae Jeong, Cosmin Paduraru, Nicolas Heess, Sumanth Dathathri, Martin Riedmiller, Timothy Mann

TL;DR

<3-5 sentence high-level summary> The paper tackles constrained reinforcement learning under model misspecification, where real-world perturbations can degrade both return and constraint satisfaction. It introduces two robust objective families, R3C and RC, within a Robust Constrained MDP (RC-MDP) framework and defines corresponding Bellman operators that converge to fixed points. These ideas are integrated into state-of-the-art continuous control learners (D4PG and DMPO) and evaluated on six Real-World RL Mujoco tasks, showing improved robustness to unseen perturbations and better constraint satisfaction, with some cases of conservatism. The work advances practical robust RL for constrained continuous control by formalizing the RC-MDP, providing stable evaluation updates, and demonstrating empirical gains across diverse perturbed environments.</p>

Abstract

Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm that mitigates this form of misspecification, and showcase its performance in multiple simulated Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

TL;DR

<3-5 sentence high-level summary> The paper tackles constrained reinforcement learning under model misspecification, where real-world perturbations can degrade both return and constraint satisfaction. It introduces two robust objective families, R3C and RC, within a Robust Constrained MDP (RC-MDP) framework and defines corresponding Bellman operators that converge to fixed points. These ideas are integrated into state-of-the-art continuous control learners (D4PG and DMPO) and evaluated on six Real-World RL Mujoco tasks, showing improved robustness to unseen perturbations and better constraint satisfaction, with some cases of conservatism. The work advances practical robust RL for constrained continuous control by formalizing the RC-MDP, providing stable evaluation updates, and demonstrating empirical gains across diverse perturbed environments.</p>

Abstract

Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm that mitigates this form of misspecification, and showcase its performance in multiple simulated Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

Paper Structure

This paper contains 27 sections, 3 theorems, 15 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Processes
  4. Continuous Control RL Algorithms
  5. Robust Constrained Optimization Framework
  6. Robust Return Robust Constraint (R3C) Objective
  7. Robust Constrained (RC) Objective
  8. Lagrange update
  9. Robust Constrained Policy Evaluation
  10. Experiments
  11. Experimental Setup
  12. Robustness Experimental Setup:
  13. Main Results
  14. Fixed Policy Sensitivity
  15. Overall performance
  16. ...and 12 more sections

Key Result

Theorem 1

Given an arbitrary return value function $V:S\rightarrow \mathbb{R}$ and an arbitrary constraint value function $V_C:S\rightarrow \mathbb{R}$, the R3C Bellman operator $\mathcal{T}^{\pi}_{R3C}:\mathbb{R}^{|S|} \rightarrow \mathbb{R}^{|S|}$ when applied iteratively to $\mathbf{V}=\langle V, V_C \rang

Figures (10)

  • Figure 1: The effect on constraint satisfaction and return as perturbations are added to cartpole, quadruped and walker for a fixed C-D4PG policy.
  • Figure 2: The holdout set performance of the baseline algorithms on DMPO variants for Cartpole with slider damping and pole mass perturbations, and Walker with thigh length perturbations (bottom row).
  • Figure 3: The holdout set performance of the baseline algorithms on D4PG variants for Cartpole with slider damping and pole mass perturbations, and Walker with thigh length perturbations (bottom row).
  • Figure 4: Learning curves of the DMPO variants for Task 4 from Table \ref{['tab:main_task_defs']} - the Walker domain with thigh length perturbations. This includes the episode return and constraint satisfaction performance (with respect to the threshold $\beta$) for the nominal model (a, c) and a holdout set (b, d) and (e) Lagrange learning performance.
  • Figure 5: The effect on constraint satisfaction and return as perturbations are added to cartpole for a fixed C-D4PG policy.
  • ...and 5 more figures

Theorems & Definitions (9)

  • Definition 1
  • Definition 2: R3C Value Function
  • Definition 3: R3C Bellman operator
  • Theorem 1
  • proof
  • Lemma 2: Lagrange derivative
  • proof
  • Theorem 3: Sup Bellman operator contraction
  • proof