Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety-Biased Policy Optimisation: Towards Hard-Constrained Reinforcement Learning via Trust Regions

Ankit Kanwar, Dominik Wagner, Luke Ong

TL;DR

SB-TRPO addresses hard safety constraints in reinforcement learning by introducing a safety-biased trust-region update that convexly combines cost and reward gradients under a KL constraint. The framework guarantees monotonic cost reduction and, when gradient directions align, rewards improve, with a beta parameter controlling the aggressiveness of safety progress. The method is analyzed via second-order approximations and a cost-biased convex combination, and validated on Safety Gymnasium where it achieves the best balance between safety and task completion relative to strong baselines, while reducing computational cost by omitting critics. This work offers a practically effective, theoretically motivated approach to hard-constrained RL, enabling steadier learning and safer policies in safety-critical applications.

Abstract

Reinforcement learning (RL) in safety-critical domains requires agents to maximise rewards while strictly adhering to safety constraints. Existing approaches, such as Lagrangian and projection-based methods, often either fail to ensure near-zero safety violations or sacrifice reward performance in the face of hard constraints. We propose Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a new trust-region algorithm for hard-constrained RL. SB-TRPO adaptively biases policy updates towards constraint satisfaction while still seeking reward improvement. Concretely, it performs trust-region updates using a convex combination of the natural policy gradients of cost and reward, ensuring a fixed fraction of optimal cost reduction at each step. We provide a theoretical guarantee of local progress towards safety, with reward improvement when gradients are suitably aligned. Experiments on standard and challenging Safety Gymnasium tasks show that SB-TRPO consistently achieves the best balance of safety and meaningful task completion compared to state-of-the-art methods.

Safety-Biased Policy Optimisation: Towards Hard-Constrained Reinforcement Learning via Trust Regions

TL;DR

SB-TRPO addresses hard safety constraints in reinforcement learning by introducing a safety-biased trust-region update that convexly combines cost and reward gradients under a KL constraint. The framework guarantees monotonic cost reduction and, when gradient directions align, rewards improve, with a beta parameter controlling the aggressiveness of safety progress. The method is analyzed via second-order approximations and a cost-biased convex combination, and validated on Safety Gymnasium where it achieves the best balance between safety and task completion relative to strong baselines, while reducing computational cost by omitting critics. This work offers a practically effective, theoretically motivated approach to hard-constrained RL, enabling steadier learning and safer policies in safety-critical applications.

Abstract

Reinforcement learning (RL) in safety-critical domains requires agents to maximise rewards while strictly adhering to safety constraints. Existing approaches, such as Lagrangian and projection-based methods, often either fail to ensure near-zero safety violations or sacrifice reward performance in the face of hard constraints. We propose Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a new trust-region algorithm for hard-constrained RL. SB-TRPO adaptively biases policy updates towards constraint satisfaction while still seeking reward improvement. Concretely, it performs trust-region updates using a convex combination of the natural policy gradients of cost and reward, ensuring a fixed fraction of optimal cost reduction at each step. We provide a theoretical guarantee of local progress towards safety, with reward improvement when gradients are suitably aligned. Experiments on standard and challenging Safety Gymnasium tasks show that SB-TRPO consistently achieves the best balance of safety and meaningful task completion compared to state-of-the-art methods.
Paper Structure (51 sections, 9 theorems, 34 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 51 sections, 9 theorems, 34 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Limitations of existing approaches.
  3. Our approach.
  4. Contributions.
  5. Related Work
  6. Primal–Dual Methods.
  7. Trust-Region and Projection-Based Methods.
  8. Reward–Cost Switching.
  9. Model-Based and Shielding Approaches.
  10. Problem Formulation
  11. Discussion.
  12. Method
  13. Safety-Biased Trust Region Update
  14. $\beta=1$ recovers CPO.
  15. Intuition for using a safety bias $\beta < 1$.
  16. ...and 36 more sections

Key Result

Theorem 1

Let $\pi_0,\pi_1,\dots$ be the sequence of policies generated by the idealised (eq:general). Then

Figures (3)

  • Figure 1: Visualisation of the adaptive convex combination $\Delta$ of $\Delta_r$ and $\Delta_c$ given by \ref{['eq:conv']} for $\epsilon\coloneqq 1.4=-\beta\cdot\langle g_c,\Delta_c\rangle$, where $\beta\coloneqq0.7$, and the special case that $\Delta_r=g_r$ and $\Delta_c=-g_c$.
  • Figure 2: Maintaining safety and ablation studies.
  • Figure 3: Safe navigation tasks of Safety Gymnasium ji2023safety (images taken from https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_navigation.html)

Theorems & Definitions (16)

  • Theorem 1
  • Theorem 2: Performance Improvement
  • Lemma 1
  • proof
  • Theorem 2
  • proof : Proof sketch
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • ...and 6 more