Marvel: Accelerating Safe Online Reinforcement Learning with Finetuned Offline Policy

TL;DR

Marvel tackles fast, safe online reinforcement learning under a safety constraint by leveraging offline safe RL to provide a warm start. It combines Value Pre-Alignment (VPA), which re-aligns the pretrained reward and cost Q-functions with the online objective using offline data, and Adaptive PID Control (aPID), which adaptively tunes the Lagrange multiplier during online finetuning. Empirical results across Bullet Safety Gym tasks show Marvel delivers higher rewards while strictly satisfying the cost bound with only a handful of online updates, and ablations confirm the necessity of both VPA and aPID. The method is compatible with diverse offline safe RL baselines, offering a practical, data-efficient path to real-world safe RL deployment.

Abstract

The high costs and risks involved in extensive environment interactions hinder the practical application of current online safe reinforcement learning (RL) methods. While offline safe RL addresses this by learning policies from static datasets, the performance therein is usually limited due to reliance on data quality and challenges with out-of-distribution (OOD) actions. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online policy learning, a direction that has yet to be fully investigated. To fill this gap, we first demonstrate that naively applying existing O2O algorithms from standard RL would not work well in the safe RL setting due to two unique challenges: \emph{erroneous Q-estimations}, resulted from offline-online objective mismatch and offline cost sparsity, and \emph{Lagrangian mismatch}, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce \textbf{Marvel}, a novel framework for O2O safe RL, comprising two key components that work in concert: \emph{Value Pre-Alignment} to align the Q-functions with the underlying truth before online learning, and \emph{Adaptive PID Control} to effectively adjust the Lagrange multipliers during online finetuning. Extensive experiments demonstrate that Marvel significantly outperforms existing baselines in both reward maximization and safety constraint satisfaction. By introducing the first policy-finetuning based framework for O2O safe RL, which is compatible with many offline and online safe RL methods, our work has the great potential to advance the field towards more efficient and practical safe RL solutions.

Figures (16)

• Figure 1: "Steps" on the x-axis represent the number of policy gradient updates (i.e., optimizer updates). For each update, the agent interacts with the environment for 3 episodes. This convention is followed in the subsequent figures. In (a), we evaluate these methods in BallCircle from Bullet Safety Gym gronauer2022bullet, with a cost limit of 20. As shown, while "Warm Start" begins with a reasonably good initial policy, it performs poorly and overly conservatively, even worse than "From Scratch" where the policy and Q-functions are initialized randomly. This implies that directly finetuning the pretrained policy and Q-functions may hinder online learning. In contrast, "Marvel" achieves impressive results, finding a policy with much higher return in just a few online steps while adhering to the cost limit. In (b), t-SNE visualization of state in the environment, reduced to 2D space. Each point represents a state, with rewards uniformly distributed across the space, while costs are sparse, appearing as isolated points or clusters, reflecting their limited association with states.
• Figure 2: Comparison of online finetuning performance after VPA with two different initial values of the Lagrange multiplier. In 'VPA w/o init', the initial value is set to 0, whereas we initialize Lagrange multipliers with a good value found empirically (0.65 in BallCircle and 0.5 in CarRun) in 'VPA w/ init'. The multiplier is then updated using the standard dual ascent method.
• Figure 3: 'aPID' refers to adaptive PID applied only during online finetuning; 'VPA' refers to VPA is only used in pre-finetuning. 'VPA+PID' uses standard PID and VPA, while 'VPA+aPID' combines VPA with adaptive PID. VPA+aPID shows the best learning efficiency, stability, and convergence.
• Figure 4: Performance comparison between Marvel and baseline methods in multiple environments. It is clear that Marvel can quickly find a high-return policy while keeping the cost below the limit.
• Figure 5: We provide experiments on more environments.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2: Contraction Property
• proof
• Lemma 1: Bounded Value Function
• proof
• proof