Safe Exploration via Policy Priors

TL;DR

This work tackles safe exploration in reinforcement learning by introducing SOOPER, a model-based algorithm that leverages pessimistic policy priors to guarantee safety while exploring through a learned world model.SOOPER reframes safety via a planning MDP with a pessimistic cost bound and a unified objective that adds intrinsic rewards for exploration and expansion, enabling efficient use of offline or simulator-trained priors.The authors prove high-probability safety throughout learning and establish a sublinear cumulative regret bound, showing convergence toward the CMDP feasible set, and validate the approach on safe RL benchmarks and real hardware.By blending conservative priors with optimistic planning, SOOPER demonstrates improved sample efficiency and safety performance under distribution shifts and realistic noise, highlighting the practical impact of prior-informed safe exploration.

Abstract

Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (e.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior if needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.

Figures (18)

• Figure 1: We denote the implicit set of safe policies in iteration $n$ by $\Pi_{<d}^n$, based on the learned model $\mathcal{F}_n$. Left: exploration-exploitation in constrained tasks may not find an optimal policy because search is limited to $\Pi_{<d}^n$. Right: expansion proactively enlarges the safe set and reaches the optimum.
• Figure 2: Performance improvement over the baseline policy and largest constraint recorded throughout learning. Among all methods, SOOPER remains safe in all tasks while consistently outperforming or being on par with the other baselines when they satisfy the constraints.
• Figure 3: Image observations for the CartpoleSwingup task. The goal is to swing the pendulum to the upright position while avoiding the range outside the vertical red lines.
• Figure 4: Learning curves of the objective and constraint when learning from offline and vision policies. SOOPER satisfies the constraints while significantly improving over the initial prior policy.
• Figure 5: Safe exploration on real hardware with SOOPER. We report the mean and standard error across five seeds of the objective and constraint measured on the real system. SOOPER learns to improve over the prior policy while satisfying the constraints throughout learning. https://anonymous.4open.science/r/sooper/docs/videos/SOOPER/timelapse.mp4

Theorems & Definitions (34)

• Definition 1
• Theorem 1: Safety guarantee
• Theorem 2: Sublinear cumulative regret
• Lemma 1
• proof
• Lemma 2
• proof
• Theorem 2: Safety guarantee
• proof
• Lemma 3