Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Optimization for Offline Safe Reinforcement Learning

Yassine Chemingui, Aryan Deshwal, Alan Fern, Thanh Nguyen-Tang, Janardhan Rao Doppa

TL;DR

This paper introduces O3SRL, a minimax framework for offline safe reinforcement learning that learns reward-maximizing policies under a cumulative cost constraint using fixed data. It combines an offline RL oracle with a no-regret online optimizer to update the Lagrange multiplier, providing convergence guarantees to the minimax solution. To address practical challenges, it offers a discrete-λ, EXP3-based approximation that avoids repeated offline policy evaluation and reduces computation, while remaining compatible with various offline RL methods. Empirical results on the DSRL benchmark show that O3SRL reliably enforces safety under tight budgets and achieves competitive rewards, outperforming several state-of-the-art baselines. The work suggests strong applicability to safety-critical domains and potential extensions to offline-to-online safe RL scenarios.

Abstract

We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.

Online Optimization for Offline Safe Reinforcement Learning

TL;DR

This paper introduces O3SRL, a minimax framework for offline safe reinforcement learning that learns reward-maximizing policies under a cumulative cost constraint using fixed data. It combines an offline RL oracle with a no-regret online optimizer to update the Lagrange multiplier, providing convergence guarantees to the minimax solution. To address practical challenges, it offers a discrete-λ, EXP3-based approximation that avoids repeated offline policy evaluation and reduces computation, while remaining compatible with various offline RL methods. Empirical results on the DSRL benchmark show that O3SRL reliably enforces safety under tight budgets and achieves competitive rewards, outperforming several state-of-the-art baselines. The work suggests strong applicability to safety-critical domains and potential extensions to offline-to-online safe RL scenarios.

Abstract

We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.
Paper Structure (32 sections, 2 theorems, 20 equations, 3 figures, 11 tables, 2 algorithms)

This paper contains 32 sections, 2 theorems, 20 equations, 3 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Related Work
  4. O3SRL Framework for Offline Constrained RL
  5. Minimax Optimization Formulation
  6. General O3SRL Framework via No-Regret Algorithms
  7. Practical Algorithms
  8. Final Approximate Algorithm for Empirical Evaluation
  9. Experiments and Results
  10. Experimental Research Questions
  11. Experimental Setup
  12. Results and Discussion
  13. Summary and Future Work
  14. Proofs for Theorem 1 and Theorem 2
  15. Extended Discussion of Related Work
  16. ...and 17 more sections

Key Result

Theorem 1

Let $(\bar{D}, \bar{\lambda})$ be the output of algorithm: osrl via no-regret. Then, $(\bar{D}, \bar{\lambda})$ is $\epsilon$-approximate equilibrium, where $\epsilon = {\epsilon_{\text{offline-RL}}}(n) + \frac{R_T(\Lambda)}{T}$.

Figures (3)

  • Figure 1: Visualization of the Bullet-Safety-Gym environments featuring different agents and tasks.
  • Figure 2: An example of a heavy-tailed reward distribution from the BallRun dataset.
  • Figure 3: Training curves of O3SRL across different tasks, showing average reward (top) and cost (bottom) over training iterations.

Theorems & Definitions (8)

  • Definition 1: $\epsilon$-approximate equilibrium
  • Definition 2: Offline RL oracle
  • Definition 3: No-regret algorithm
  • Theorem 1
  • Definition 4
  • Theorem 2
  • proof : Proof of \ref{['theorem: main']}
  • proof : Proof of \ref{['theorem: error rate for exp3']}