Skill-based Safe Reinforcement Learning with Risk Planning

TL;DR

The paper tackles safe reinforcement learning in high-cost environments by leveraging offline demonstrations to learn high-level skills and a risk predictor trained with Positive-Unlabeled learning. It introduces Safe Skill Planning (SSkP), a two-stage method: (i) extract latent skills and build a risk predictor $P_\zeta(c|s_t,\boldsymbol{z}_t)$ via PU learning, and (ii) perform risk planning to iteratively select safe skills during online training, followed by policy learning with Soft Actor-Critic under a KL penalty to the skill prior. The approach yields a risk-aware online RL agent that achieves higher rewards with fewer safety violations than state-of-the-art baselines (CPQ, SMBPO, Recovery RL) across four MuJoCo-based robotics environments. This work demonstrates that combining offline skill representations with probabilistic risk evaluation and iterative planning can significantly reduce unsafe exploration while maintaining strong task performance, with practical impact for real-world robotic learning.

Abstract

Safe Reinforcement Learning (Safe RL) aims to ensure safety when an RL agent conducts learning by interacting with real-world environments where improper actions can induce high costs or lead to severe consequences. In this paper, we propose a novel Safe Skill Planning (SSkP) approach to enhance effective safe RL by exploiting auxiliary offline demonstration data. SSkP involves a two-stage process. First, we employ PU learning to learn a skill risk predictor from the offline demonstration data. Then, based on the learned skill risk predictor, we develop a novel risk planning process to enhance online safe RL and learn a risk-averse safe policy efficiently through interactions with the online RL environment, while simultaneously adapting the skill risk predictor to the environment. We conduct experiments in several benchmark robotic simulation environments. The experimental results demonstrate that the proposed approach consistently outperforms previous state-of-the-art safe RL methods.

Figures (6)

• Figure 1: The framework of the proposed method, SSkP, which learns a skill risk predictor from the offline demonstration data and then deploys it to enhance online safe RL through risk planning. During the skill risk predictor learning stage, SSkP assembles PU data and trains a decision risk predictor $P_\zeta(c|s_t,z_t)$ based on a skill model, which produces skill prior $q_\psi(\cdot|s_t)$ and skill decoder $p_\nu(\textbf{a}_t|z_t)$. In the online safe policy learning stage, a risk planning process is deployed to generate and choose safer skill decisions based on the skill risk predictor $P_\zeta(c|s_t,z_t)$. The generated skill $z_t$ is decoded by the skill decoder $p_\nu(\textbf{a}_t|z_t)$ into an action sequence $\textbf{a}_t$ to interact with the online environment. Rewards are collected from these online interactions to learn the safe skill policy $\pi_\theta(z_t|s_t)$.
• Figure 2: The four environments employed in the experiments are displayed from left to right: Ant, Cheetah, Hopper, Humanoid. The figures present instances of failure in each environment where safety constraints are violated.
• Figure 3: This figure presents the performance of each comparison method in terms of the average episode reward vs. the total number of safety violations encountered during online training within a fixed total number of timesteps on all four environments: Ant, Cheetah, Hopper, and Humanoid. The results represent the averages over three runs, with the shadow indicating the standard deviations.
• Figure 4: The ablation study results in two environments: Ant and Hopper by comparing three methods: SSkP---the proposed approach; SSkP-NP---the variant that replaces risk planning with a naive planning process; and SSkP-w/o-RP---the variant that drops risk predictor and risk planning from SSkP. Each plot displays the average reward vs. the total number of safety violations encountered during online training within a fixed total number of timesteps. The results are averages of three runs.
• Figure 5: Risk prediction probability changes, $\nabla\bar{p}_i=\bar{p}_i-\bar{p}_0$, along the planning iteration number $i$ from the initial average risk prediction probability $\bar{p}_0$. The results are the averages computed with the risk planning procedure on 100 randomly sampled states $s_t$.