Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Honghao Wei, Xiyue Peng, Arnob Ghosh, Xin Liu

Abstract

We propose WSAC (Weighted Safe Actor-Critic), a novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed as a two-player Stackelberg game to optimize a refined objective function. The actor optimizes the policy against two adversarially trained value critics with small importance-weighted Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, we demonstrate that when the actor employs a no-regret optimization oracle, WSAC achieves a number of guarantees: (i) For the first time in the safe offline RL setting, we establish that WSAC can produce a policy that outperforms any reference policy while maintaining the same level of safety, which is critical to designing a safe algorithm for offline RL. (ii) WSAC achieves the optimal statistical convergence rate of $1/\sqrt{N}$ to the reference policy, where $N$ is the size of the offline dataset. (iii) We theoretically show that WSAC guarantees a safe policy improvement across a broad range of hyperparameters that control the degree of pessimism, indicating its practical robustness. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-art safe offline RL algorithms in several continuous control environments. WSAC outperforms all baselines across a range of tasks, supporting the theoretical results.

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Abstract

We propose WSAC (Weighted Safe Actor-Critic), a novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed as a two-player Stackelberg game to optimize a refined objective function. The actor optimizes the policy against two adversarially trained value critics with small importance-weighted Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, we demonstrate that when the actor employs a no-regret optimization oracle, WSAC achieves a number of guarantees: (i) For the first time in the safe offline RL setting, we establish that WSAC can produce a policy that outperforms any reference policy while maintaining the same level of safety, which is critical to designing a safe algorithm for offline RL. (ii) WSAC achieves the optimal statistical convergence rate of to the reference policy, where is the size of the offline dataset. (iii) We theoretically show that WSAC guarantees a safe policy improvement across a broad range of hyperparameters that control the degree of pessimism, indicating its practical robustness. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-art safe offline RL algorithms in several continuous control environments. WSAC outperforms all baselines across a range of tasks, supporting the theoretical results.
Paper Structure (30 sections, 12 theorems, 53 equations, 4 figures, 5 tables, 2 algorithms)

This paper contains 30 sections, 12 theorems, 53 equations, 4 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Constrained Markov Decision Process
  5. Function Approximation
  6. Offline RL
  7. Actor-Critic with Importance Weighted Bellman Error
  8. Theoretical Analysis of WSAC
  9. Main Algorithm
  10. Theoretical Guarantees
  11. A Safe Robust Policy Improvement
  12. Experiments
  13. WSAC-Practical Implementation
  14. Simulations
  15. Conclusion
  16. ...and 15 more sections

Key Result

Lemma 1

Define the $\ell_\infty$ single policy concentrability RasZhuMa_21 as $C_{\ell_\infty}^\pi = \Vert d^\pi / \mu\Vert_{\infty}$ and the Bellman-consistent single-policy concentrability XieCheJia_21 as $C_{Bellman}^\pi = \max_{f\in\mathcal{F}}\frac{\Vert f-\mathcal{T}^\pi f\Vert_{2,d^\pi}^2 }{ \Vert f-

Figures (4)

  • Figure 1: Comparison between WSAC and the behavior policy in the tabular case. The behavior policy is a mixture of the optimal policy and a random policy, with the mixture percentage representing the proportion of the optimal policy. The cost threshold is set to 0.1. We observe that WSAC consistently ensures a safely improved policy across various scenarios, even when the behavior policy is not safe.
  • Figure 2: BallCircle and CarCircle (left), PointButton (medium), PointPush(right) .
  • Figure 3: The moving average of evaluation results is recorded every $500$ training steps, with each result representing the average over $20$ evaluation episodes and three random seeds. A cost threshold $1$ is applied, with any normalized cost below 1 considered safe.
  • Figure 4: Sensitivity Analysis of Hyperparameters in the Tabular Case. The left figure illustrates tests conducted with various $\beta$ values (For the sake of discussion, we denote $\beta = \beta_r = \beta_c$) with $\lambda = [0,2]$, while the right figure presents tests across different ranges of $\lambda$ with $\beta_r = \beta_c = 2.0$.

Theorems & Definitions (28)

  • Remark 3.1
  • Definition 3.3: $\ell_2$ Concentrability
  • Remark 3.4
  • Lemma 1: Restate Proposition $2.1$ in ZhuRasJia_23
  • Remark 3.5
  • Theorem 4.1
  • Definition 5.1: No-regret policy optimization oracle
  • Lemma 2
  • Theorem 5.2: Main Theorem
  • Remark 5.3
  • ...and 18 more