Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A universal policy wrapper with guarantees

Anton Bolychev, Georgiy Malaniya, Grigory Yaremenko, Anastasia Krasnaya, Pavel Osinenko

TL;DR

The paper presents CALF-Wrapper, a universal runtime policy wrapper that guarantees goal-reaching by dynamically switching between a high-performing base RL policy and a safe fallback policy guided by the base policy's value function. The approach provides formal guarantees (Theorems 1 and 2) without requiring online constrained optimization or explicit system dynamics, and it is implemented as a lightweight Gymnasium wrapper that can be plugged into existing RL workflows. Theoretical analysis shows that the wrapper eventually relies on the fallback policy, inheriting its goal-reaching properties, while empirical results on Pendulum-v1 and CartPoleSwingup demonstrate controllable trade-offs between safety and performance via Conservative, Balanced, and Brave modes. The work offers a practical, deployable method to inject safety guarantees into a broad class of RL systems, expanding the applicability of RL to safety-critical settings.

Abstract

We introduce a universal policy wrapper for reinforcement learning agents that ensures formal goal-reaching guarantees. In contrast to standard reinforcement learning algorithms that excel in performance but lack rigorous safety assurances, our wrapper selectively switches between a high-performing base policy -- derived from any existing RL method -- and a fallback policy with known convergence properties. Base policy's value function supervises this switching process, determining when the fallback policy should override the base policy to ensure the system remains on a stable path. The analysis proves that our wrapper inherits the fallback policy's goal-reaching guarantees while preserving or improving upon the performance of the base policy. Notably, it operates without needing additional system knowledge or online constrained optimization, making it readily deployable across diverse reinforcement learning architectures and tasks.

A universal policy wrapper with guarantees

TL;DR

The paper presents CALF-Wrapper, a universal runtime policy wrapper that guarantees goal-reaching by dynamically switching between a high-performing base RL policy and a safe fallback policy guided by the base policy's value function. The approach provides formal guarantees (Theorems 1 and 2) without requiring online constrained optimization or explicit system dynamics, and it is implemented as a lightweight Gymnasium wrapper that can be plugged into existing RL workflows. Theoretical analysis shows that the wrapper eventually relies on the fallback policy, inheriting its goal-reaching properties, while empirical results on Pendulum-v1 and CartPoleSwingup demonstrate controllable trade-offs between safety and performance via Conservative, Balanced, and Brave modes. The work offers a practical, deployable method to inject safety guarantees into a broad class of RL systems, expanding the applicability of RL to safety-critical settings.

Abstract

We introduce a universal policy wrapper for reinforcement learning agents that ensures formal goal-reaching guarantees. In contrast to standard reinforcement learning algorithms that excel in performance but lack rigorous safety assurances, our wrapper selectively switches between a high-performing base policy -- derived from any existing RL method -- and a fallback policy with known convergence properties. Base policy's value function supervises this switching process, determining when the fallback policy should override the base policy to ensure the system remains on a stable path. The analysis proves that our wrapper inherits the fallback policy's goal-reaching guarantees while preserving or improving upon the performance of the base policy. Notably, it operates without needing additional system knowledge or online constrained optimization, making it readily deployable across diverse reinforcement learning architectures and tasks.
Paper Structure (19 sections, 8 theorems, 24 equations, 3 figures, 1 algorithm)

This paper contains 19 sections, 8 theorems, 24 equations, 3 figures, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Key Contributions
  3. Related Work on Safe Reinforcement Learning
  4. PROBLEM STATEMENT
  5. Markov Decision Process
  6. Policies and Value Functions
  7. Goal Set and Goal Reaching Property
  8. PROPOSED APPROACH
  9. Action Selection Mechanism and Its Benefits
  10. Critic-Based Decision Rule
  11. Intuition Behind the Mechanism
  12. Adjustable Performance-Safety Balance
  13. Seamless Integration
  14. Rigorous Stability Guarantees
  15. Theoretical Analysis
  16. ...and 4 more sections

Key Result

Theorem 1

Suppose that $\bar{v} = \sup_{s \in \mathbb S} \hat{v}^{\pi_{b}}(s) < \infty$. Then policy $\pi_{t}$ satisfies the ${\varepsilon}$-improbable goal reaching property from Definition dfn:eta_improbable.

Figures (3)

  • Figure 1: Comparative analysis of CALF-Wrapper modes on Pendulum-v1 and CartPoleSwingup environments: Conservative ($p_{\mathrm{relax}} = 0$), Balanced ($p_{\mathrm{relax}} = 0.5$), and Brave ($p_{\mathrm{relax}} = 0.95$) modes against base and fallback policies. Box plots show mean cumulative rewards with standard deviations across 30 trials.
  • Figure :
  • Figure :

Theorems & Definitions (16)

  • Definition 1: Goal Set
  • Definition 2: ${\varepsilon}$-Improbable Goal Reaching Property
  • Definition 3: Uniform ${\varepsilon}$-Improbable Goal Reaching
  • Definition 4
  • Theorem 1
  • proof
  • Definition 5
  • Theorem 2
  • proof
  • Proposition 1
  • ...and 6 more