Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art

TL;DR

This work tackles safe continual reinforcement learning (COSRL) in nonstationary environments, where distribution shifts challenge both performance and safety. It surveys state-of-the-art COSRL methods, classifying them by adaptation mechanisms (passive, reactive, quick, proactive) and analyzes safety constraint formulations across NS frameworks such as CMDP, NSMDP, POMDP, HM-MDP, BAMDP, and DP-MDP. The authors provide a taxonomy of constraints (predefined, partially adjustable, adjustable) and discuss data-driven, time-dependent, and variational budgets, highlighting the predominance of soft guarantees and the need for context-aware hard constraints. Key contributions include a structured synthesis of learning, adaptation, and safety mechanisms, plus guidance on future research directions like context-based meta-learning and proactive constraint modeling to enable safer deployment of COSRL in real-world nonstationary settings.

Abstract

This work provides a state-of-the-art survey of continual safe online reinforcement learning (COSRL) methods. We discuss theoretical aspects, challenges, and open questions in building continual online safe reinforcement learning algorithms. We provide the taxonomy and the details of continual online safe reinforcement learning methods based on the type of safe learning mechanism that takes adaptation to nonstationarity into account. We categorize safety constraints formulation for online reinforcement learning algorithms, and finally, we discuss prospects for creating reliable, safe online learning algorithms. Keywords: safe RL in nonstationary environments, safe continual reinforcement learning under nonstationarity, HM-MDP, NSMDP, POMDP, safe POMDP, constraints for continual learning, safe continual reinforcement learning review, safe continual reinforcement learning survey, safe continual reinforcement learning, safe online learning under distribution shift, safe continual online adaptation, safe reinforcement learning, safe exploration, safe adaptation, constrained Markov decision processes, safe reinforcement learning, partially observable Markov decision process, safe reinforcement learning and hidden Markov decision processes, Safe Online Reinforcement Learning, safe online reinforcement learning, safe online reinforcement learning, safe meta-learning, safe meta-reinforcement learning, safe context-based reinforcement learning, formulating safety constraints for continual learning