Constrained Reinforcement Learning Under Model Mismatch
Zhongchang Sun, Sihong He, Fei Miao, Shaofeng Zou
TL;DR
This work tackles robust constrained reinforcement learning under model mismatch by formulating a constrained MDP with an uncertainty set of transition kernels and aiming to maximize worst-case reward while maintaining a robust constraint. It introduces Robust Constrained Policy Optimization (RCPO), a primal method that alternates a robust policy-improvement step with a constraint-projection step, and generalizes the performance-difference principle to robust MDPs. Theoretical guarantees bound per-update worst-case reward improvement and constraint violation, accommodating continuous state spaces, along with practical approximations for scalable implementation. Empirical results on tabular and deep tasks under model uncertainty validate RCPO's ability to deliver monotone robust improvement and constraint satisfaction, demonstrating its potential for safe deployment in real-world constrained RL problems.
Abstract
Existing studies on constrained reinforcement learning (RL) may obtain a well-performing policy in the training environment. However, when deployed in a real environment, it may easily violate constraints that were originally satisfied during training because there might be model mismatch between the training and real environments. To address the above challenge, we formulate the problem as constrained RL under model uncertainty, where the goal is to learn a good policy that optimizes the reward and at the same time satisfy the constraint under model mismatch. We develop a Robust Constrained Policy Optimization (RCPO) algorithm, which is the first algorithm that applies to large/continuous state space and has theoretical guarantees on worst-case reward improvement and constraint violation at each iteration during the training. We demonstrate the effectiveness of our algorithm on a set of RL tasks with constraints.