Safe and Robust Reinforcement Learning: Principles and Practice

TL;DR

The paper addresses the safety and robustness challenges in deploying reinforcement learning (RL) by clarifying working definitions of safe and robust RL and surveying a wide range of algorithmic, ethical, and practical approaches. It categorises methods into robust and constrained optimisation (e.g., $J_r^{\pi}$, $J_{c_i}^{\pi}$ with $\epsilon_i$), explores leveraging uncertainty, adversarial training, and safe exploration, and discusses incorporating external data (trajectory datasets, simulators, human knowledge) and human-in-the-loop strategies. It also situates safe/robust RL within related areas (control theory, transfer/meta-learning, sim-to-real) and highlights ethical considerations such as reward misspecification and transparency, offering a practical, practitioner-friendly checklist for deployment. The work aims to provide a foundational resource that guides researchers and policymakers toward responsible real-world RL systems, combining theoretical rigor with actionable guidelines. The proposed checklist and definitions help standardise practice across diverse application domains, enabling safer and more reliable RL deployments with measurable safety and robustness guarantees.

Abstract

Reinforcement Learning (RL) has shown remarkable success in solving relatively complex tasks, yet the deployment of RL systems in real-world scenarios poses significant challenges related to safety and robustness. This paper aims to identify and further understand those challenges thorough the exploration of the main dimensions of the safe and robust RL landscape, encompassing algorithmic, ethical, and practical considerations. We conduct a comprehensive review of methodologies and open problems that summarizes the efforts in recent years to address the inherent risks associated with RL applications. After discussing and proposing definitions for both safe and robust RL, the paper categorizes existing research works into different algorithmic approaches that enhance the safety and robustness of RL agents. We examine techniques such as uncertainty estimation, optimisation methodologies, exploration-exploitation trade-offs, and adversarial training. Environmental factors, including sim-to-real transfer and domain adaptation, are also scrutinized to understand how RL systems can adapt to diverse and dynamic surroundings. Moreover, human involvement is an integral ingredient of the analysis, acknowledging the broad set of roles that humans can take in this context. Importantly, to aid practitioners in navigating the complexities of safe and robust RL implementation, this paper introduces a practical checklist derived from the synthesized literature. The checklist encompasses critical aspects of algorithm design, training environment considerations, and ethical guidelines. It will serve as a resource for developers and policymakers alike to ensure the responsible deployment of RL systems in many application domains.

Figures (5)

• Figure 1: Overview of the reinforcement learning setting (in black) with relevant components (coloured) to the safe and robust RL introduced in Sec.\ref{['sec02:optimisation']} to Sec.\ref{['sec02:human-in-the-loop']}
• Figure 2: Overview of the reinforcement learning setting.
• Figure 3: Categories of constraints.
• Figure 4: The figure organises safe reinforcement learning literature in chronological order, highlighted in colour-coded boxes for major categories and black boxes for sub-categories. Black arrows show inter-paper inspiration across categories.
• Figure 5: The figure organises robust reinforcement learning literature in chronological order, highlighted in colour-coded boxes for major categories and black boxes for sub-categories.

Theorems & Definitions (2)

• Definition 1: Robust RL
• Definition 2: Safe RL