Resilient Constrained Reinforcement Learning
Dongsheng Ding, Zhengyan Huan, Alejandro Ribeiro
TL;DR
This work tackles constrained MDPs with unknown constraint specifications by introducing a relaxation mechanism for constraints via a cost function and defining a resilient equilibrium that governs the trade-off between reward maximization and constraint satisfaction. It develops a tractable regularized CMDP formulation and two provably convergent policy-search algorithms, ResPG-PD and ResOPG-PD, that jointly optimize policy and constraint relaxation. Theoretical results establish monotonicity and concavity of the relaxed value function and connect to duality through geometric multipliers, yielding non-asymptotic convergence guarantees. Empirical results in robotic monitoring and resource-like settings show that resilience enables robust adaptation to infeasible or uncertain constraints, sustaining performance when nominal constraints cannot be met.
Abstract
We study a class of constrained reinforcement learning (RL) problems in which multiple constraint specifications are not identified before training. It is challenging to identify appropriate constraint specifications due to the undefined trade-off between the reward maximization objective and the constraint satisfaction, which is ubiquitous in constrained decision-making. To tackle this issue, we propose a new constrained RL approach that searches for policy and constraint specifications together. This method features the adaptation of relaxing the constraint according to a relaxation cost introduced in the learning objective. Since this feature mimics how ecological systems adapt to disruptions by altering operation, our approach is termed as resilient constrained RL. Specifically, we provide a set of sufficient conditions that balance the constraint satisfaction and the reward maximization in notion of resilient equilibrium, propose a tractable formulation of resilient constrained policy optimization that takes this equilibrium as an optimal solution, and advocate two resilient constrained policy search algorithms with non-asymptotic convergence guarantees on the optimality gap and constraint satisfaction. Furthermore, we demonstrate the merits and the effectiveness of our approach in computational experiments.