Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guardian: Decoupling Exploration from Safety in Reinforcement Learning

Kaitong Cai, Jusheng Zhang, Jing Yang, Keze Wang

TL;DR

This work addresses stability and safety in hybrid offline-online RL by decoupling reward-seeking learning from safety enforcement. It introduces RLPD-GX, comprising a free-running Learner and a projection-based Guardian, plus dynamic curricula (DTS/DSS) to smooth offline-online data mixing. The Guarded Bellman Operator is proven to be a $\gamma$-contraction, ensuring convergence to a safe value function, while empirical results on Atari-100k show state-of-the-art performance with stronger safety guarantees. The approach demonstrates that decoupled safety enforcement is a general, effective paradigm for reconciling exploration and safety in reinforcement learning.

Abstract

Hybrid offline--online reinforcement learning (O2O RL) promises both sample efficiency and robust exploration, but suffers from instability due to distribution shift between offline and online data. We introduce RLPD-GX, a framework that decouples policy optimization from safety enforcement: a reward-seeking learner explores freely, while a projection-based guardian guarantees rule-consistent execution and safe value backups. This design preserves the exploratory value of online interactions without collapsing to conservative policies. To further stabilize training, we propose dynamic curricula that gradually extend temporal horizons and anneal offline--online data mixing. We prove convergence via a contraction property of the guarded Bellman operator, and empirically show state-of-the-art performance on Atari-100k, achieving a normalized mean score of 3.02 (+45\% over prior hybrid methods) with stronger safety and stability. Beyond Atari, ablations demonstrate consistent gains across safety-critical and long-horizon tasks, underscoring the generality of our design. Extensive and comprehensive results highlight decoupled safety enforcement as a simple yet principled route to robust O2O RL, suggesting a broader paradigm for reconciling exploration and safety in reinforcement learning.

Guardian: Decoupling Exploration from Safety in Reinforcement Learning

TL;DR

This work addresses stability and safety in hybrid offline-online RL by decoupling reward-seeking learning from safety enforcement. It introduces RLPD-GX, comprising a free-running Learner and a projection-based Guardian, plus dynamic curricula (DTS/DSS) to smooth offline-online data mixing. The Guarded Bellman Operator is proven to be a -contraction, ensuring convergence to a safe value function, while empirical results on Atari-100k show state-of-the-art performance with stronger safety guarantees. The approach demonstrates that decoupled safety enforcement is a general, effective paradigm for reconciling exploration and safety in reinforcement learning.

Abstract

Hybrid offline--online reinforcement learning (O2O RL) promises both sample efficiency and robust exploration, but suffers from instability due to distribution shift between offline and online data. We introduce RLPD-GX, a framework that decouples policy optimization from safety enforcement: a reward-seeking learner explores freely, while a projection-based guardian guarantees rule-consistent execution and safe value backups. This design preserves the exploratory value of online interactions without collapsing to conservative policies. To further stabilize training, we propose dynamic curricula that gradually extend temporal horizons and anneal offline--online data mixing. We prove convergence via a contraction property of the guarded Bellman operator, and empirically show state-of-the-art performance on Atari-100k, achieving a normalized mean score of 3.02 (+45\% over prior hybrid methods) with stronger safety and stability. Beyond Atari, ablations demonstrate consistent gains across safety-critical and long-horizon tasks, underscoring the generality of our design. Extensive and comprehensive results highlight decoupled safety enforcement as a simple yet principled route to robust O2O RL, suggesting a broader paradigm for reconciling exploration and safety in reinforcement learning.
Paper Structure (37 sections, 1 theorem, 30 equations, 6 figures, 5 tables, 1 algorithm)

This paper contains 37 sections, 1 theorem, 30 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. MDPs in a Hybrid Data Regime
  4. Safety as a State-Action Constraint
  5. The Maximum Entropy Learning Objective
  6. Methodology: Decoupling Learning from Safety Enforcement
  7. System Architecture and Data Flow
  8. The Learner: Principled Optimization on Heterogeneous Data
  9. (a) Dynamic Temporal Sampling (DTS)
  10. (b) Dynamic Symmetric Sampling (DSS)
  11. Guarded Backups for Consistent Value Learning
  12. Theoretical Foundation: Convergence of Guarded Value Iteration
  13. Implication
  14. Experiments
  15. Main Comparisons and Analyses
  16. ...and 22 more sections

Key Result

Theorem 1

The operator $\mathcal{T}_{\Pi}$ is a $\gamma$-contraction in the max norm $\|\cdot\|_{\infty}$.

Figures (6)

  • Figure 1: Architecture of RLPD-GX. A Learner explores freely, while a projection-based Guardian ensures safe execution and guarded value backups. Dynamic sampling (DTS/DSS) with OOD regularization stabilizes hybrid offline--online learning, enabling safe yet exploratory policy updates.
  • Figure 2: Efficacy of the Guardian mechanism. Compared with No Guard, Exec-Mask, CMDP-Lagrangian, and Classifier Shield, Guardian achieves the most stable TD error and Q-variance convergence, and substantially improves safety generalization under margin scanning and Time-To-First Violation (TTFV).
  • Figure 3: Exploration efficiency comparison. Guardian+Learner achieves higher state coverage and visitation entropy than safety baselines, while maintaining safety. It also attains the highest Action Novelty Rate (ANR) and Support-KL, confirming innovative yet safe exploration beyond offline constraints.
  • Figure 4: ready to fill
  • Figure 5: ready to fill
  • ...and 1 more figures

Theorems & Definitions (3)

  • Definition 1: Guarded Bellman Operator
  • Theorem 1: Contraction
  • proof : Proof Sketch