Long-term Safe Reinforcement Learning with Binary Feedback

TL;DR

This work addresses safe reinforcement learning for CMDPs with binary safety feedback under unknown stochastic transitions, aiming to guarantee long-term safety across episodes. The authors propose LoBiSaRL, which combines GLM-based confidence bounds for the safety function with Lipschitz-based bounds to produce a pessimistic, yet informative, lower bound on safety. By formulating a constrained policy optimization and applying a Lagrangian approach, LoBiSaRL balances reward maximization with the need to maintain viable safe actions through horizon $[t,T]$ with high probability. Theoretical results establish a high-probability long-term safety guarantee, and experiments in a grid-world demonstrate safer behavior compared to baselines, albeit sometimes at some cost to reward. Overall, the method provides a principled framework for safe RL with binary feedback in uncertain environments, with potential for further performance improvements in reward while preserving safety guarantees.

Abstract

Safety is an indispensable requirement for applying reinforcement learning (RL) to real problems. Although there has been a surge of safe RL algorithms proposed in recent years, most existing work typically 1) relies on receiving numeric safety feedback; 2) does not guarantee safety during the learning process; 3) limits the problem to a priori known, deterministic transition dynamics; and/or 4) assume the existence of a known safe policy for any states. Addressing the issues mentioned above, we thus propose Long-term Binaryfeedback Safe RL (LoBiSaRL), a safe RL algorithm for constrained Markov decision processes (CMDPs) with binary safety feedback and an unknown, stochastic state transition function. LoBiSaRL optimizes a policy to maximize rewards while guaranteeing a long-term safety that an agent executes only safe state-action pairs throughout each episode with high probability. Specifically, LoBiSaRL models the binary safety function via a generalized linear model (GLM) and conservatively takes only a safe action at every time step while inferring its effect on future safety under proper assumptions. Our theoretical results show that LoBiSaRL guarantees the long-term safety constraint, with high probability. Finally, our empirical results demonstrate that our algorithm is safer than existing methods without significantly compromising performance in terms of reward.

Figures (3)

• Figure 1: Even if safety is guaranteed at time $t$ based on the instantaneous evaluation, safe behavior may not exist a few steps ahead. This paper requires an agent to guarantee long-term safety (i.e., constraint satisfaction from the time the current time step $t$ to the terminal time step $T$) in CMDPs with stochastic state transition and binary safety feedback.
• Figure 2: (a) Bounds by Lipschitz continuity for the conservative policy. (b) In the early phase of training, the lower bound of the safety linear predictor at time $t$ is typically characterized by the Lipschitz continuity, which decreases depending on the $x_1, x_2, \ldots, x_{t-1}$. Depending on the safety margin at time $t$, we need to control $x_{t}, x_{t+1}, \ldots, x_T$ for ensuring future safety. (c) As the training proceeds, the lower bound of the safety linear predictor can potentially be characterized by the GLMs, and the safety margin may increase.
• Figure 3: Example reward, binary safety, and value functions. In this paper, we consider a safe RL problem with binary safety feedback; thus, there is an unsafe region (the white region in the (c)) where the agent is not allowed to visit.

Theorems & Definitions (25)

• Remark 1
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5
• Corollary 1
• Theorem 1
• Lemma 6
• proof