Off-Policy Primal-Dual Safe Reinforcement Learning

TL;DR

This work tackles the instability of off-policy primal-dual safe reinforcement learning caused by cost underestimation in constrained MDPs. It introduces CAL, a two-ingredient framework: Conservative Policy Optimization uses an ensemble-derived upper confidence bound $Q_c^{\text{UCB}}$ to guard against underestimating costs, and Local Policy Convexification via an Augmented Lagrangian with gradient rectification to stabilize learning and reduce estimation uncertainty near local optima. Theoretical analysis and extensive experiments on Safety-Gym, MuJoCo, and a real-world semi-batch advertising task show improved sample efficiency and reduced constraint violations while achieving competitive asymptotic rewards. These results suggest CAL enables safer, more data-efficient off-policy safe RL suitable for safety-critical and large-scale applications.

Abstract

Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

Figures (8)

• Figure 1: Visualization of a part of the policy space, where deeper color indicates higher reward in the feasible (constraint-satisfying) area.
• Figure 2: Results of SAC-Lag and its conservative version (denoted as "UCB" in the legend) on two example tasks. The horizontal dashed line represents the constraint threshold in the cost plot.
• Figure 3: Results of applying local policy convexification on two example tasks, averaged over $6$ random seeds. The first two columns show the reward/cost learning curves, where the solid lines are the mean and the shadowed regions are the standard deviation. The third column shows the training cost of different timestep intervals by using box-and-whisker plots where boxes showing median, $75\%$ ($q_3$) and $25\%$ ($q_1$) quarterlies of the distributions respectively, whiskers depicting the error bounds computed as $1.5(q_3-q_1)$, as well as outliers lying outside the whisker intervals. The last column shows the trend of the standard deviation of the cost value ensemble during training, as well as the difference between the UCB cost value and its oracle value during training (where greater than $0$ implies overestimation, otherwise implies underestimation).
• Figure 4: Comparisons with off-policy baselines on Safety-Gym (top half) and velocity-constrained MuJoCo (bottom half).
• Figure 5: Ablation Studies of $k$ (top half) and $c$ (bottom half).

Theorems & Definitions (4)

• Proposition 1
• proof
• proof
• Lemma 1