Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe Reinforcement Learning with Preference-based Constraint Inference

Chenglin Li, Guangchun Ruan, Hua Geng

Abstract

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which is not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify the popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy are deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms the state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, which has great potential in a range of safety-critical applications.

Safe Reinforcement Learning with Preference-based Constraint Inference

Abstract

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which is not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify the popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy are deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms the state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, which has great potential in a range of safety-critical applications.
Paper Structure (63 sections, 5 theorems, 45 equations, 8 figures, 11 tables, 2 algorithms)

This paper contains 63 sections, 5 theorems, 45 equations, 8 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Safe RL with Unknown Constraints
  4. Learning Constraints from Preference
  5. Constraint Inference with Dead Zone
  6. Cost function learning
  7. Theoretical Analysis of Dead Zone
  8. Loss Design based on Signal-to-Noise Ratio
  9. Two-stage Training Strategy
  10. Constraint Learning Details
  11. Offline Pre-training
  12. Online Finetuning
  13. Preference-based Constrained Reinforcement Learning (PbCRL)
  14. Convergence Analysis of PbCRL
  15. Experiments
  16. ...and 48 more sections

Key Result

Lemma 3.1

For any unsafe trajectory $\tau$ with safety label $\epsilon=0$ and estimated cost $\hat{C}(\tau)$, the gradient provided by the Dead Zone loss $\mathcal{L}_{safe}^{DZ}$ in Eq. eq:safeloss is strictly more negative than that provided by the original loss $\mathcal{L}_{safe}$ in Eq. eq:safeloss0:

Figures (8)

  • Figure 1: Cost distributions: (a) Ground truth: heavy-tailed with expectation (solid line) exceeding safety threshold (dashed line), indicating unsafe. (b) BT model: symmetric with underestimated expectation below the threshold, leading to falsely believe the constraint is satisfied (Type II Error). (c) Our model: similar heavy-tailed distribution, better approximate expectation of ground truth cost.
  • Figure 2: Average episode return and cost of four algorithms on robotic navigation tasks. Dashed lines are safety thresholds.
  • Figure 3: Locomotion Environments in Safety Gymnasium
  • Figure 4: Navigation Environments in Safety Gymnasium
  • Figure 5: Autonomous Driving Environments
  • ...and 3 more figures

Theorems & Definitions (13)

  • Lemma 3.1
  • proof
  • Theorem 3.2
  • proof
  • Corollary 3.3
  • proof
  • Theorem 5.2
  • proof
  • proof
  • proof
  • ...and 3 more