Offline Safe Policy Optimization From Heterogeneous Feedback

TL;DR

This work tackles safe policy optimization in offline preference-based reinforcement learning for continuous control by introducing Offline Safe POHF and the PreSa algorithm. PreSa directly learns policies from heterogeneous offline feedback—pairwise preferences over rewards and binary safety labels—without explicitly modeling rewards or costs or performing a separate constrained RL stage, using a Lagrangian formulation to balance reward maximization with safety. The authors formalize a safety alignment component, a preference alignment component, and their integration into a unified constrained optimization, avoiding reward/cost model learning. Empirical results on synthetic and real human feedback across SafetyGym, BulletGym, and MetaDrive show that PreSa achieves robust safety with competitive or superior rewards compared to offline safe RL baselines that require ground-truth rewards/costs. This approach enables safer, data-efficient offline policy learning in real-world, high-stakes control domains like autonomous driving and robotics.

Abstract

Offline Preference-based Reinforcement Learning (PbRL) learns rewards and policies aligned with human preferences without the need for extensive reward engineering and direct interaction with human annotators. However, ensuring safety remains a critical challenge across many domains and tasks. Previous works on safe RL from human feedback (RLHF) first learn reward and cost models from offline data, then use constrained RL to optimize a safe policy. While such an approach works in the contextual bandits settings (LLMs), in long horizon continuous control tasks, errors in rewards and costs accumulate, leading to impairment in performance when used with constrained RL methods. To address these challenges, (a) instead of indirectly learning policies (from rewards and costs), we introduce a framework that learns a policy directly based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments; (b) we propose \textsc{PreSa} (Preference and Safety Alignment), a method that combines preference learning module with safety alignment in a constrained optimization problem. This optimization problem is solved within a Lagrangian paradigm that directly learns reward-maximizing safe policy \textit{without explicitly learning reward and cost models}, avoiding the need for constrained RL; (c) we evaluate our approach on continuous control tasks with both synthetic and real human feedback. Empirically, our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines, and offline safe RL approaches with ground-truth reward and cost.

Figures (10)

• Figure 1: Offline Safe POHF versus Offline PbRL for control tasks. Besides pairwise preference between agent's trajectory segments, the dataset of Offline Safe POHF additionally includes binary safety labels of each segment, which is used to align the policy with implicit safety constraints.
• Figure 2: Ratio of safe agents learned by different approaches.
• Figure 3: Performance of Offline Safe POHF methods across varying trajectory segment lengths, dataset sizes.
• Figure 4: Visualization of normalized reward and cost for each task within BulletGym domain. The dotted blue vertical lines mark the cost threshold of $1$. Each round dot represents a task, where green dots indicate tasks meeting safety constraints, and red dots indicate tasks with constraint violations.
• Figure 5: Visualization of agents and tasks in SafetyGym, BulletGym, and MetaDrive.

Theorems & Definitions (5)

• definition 1: Offline Safe Policy Optimization from Heterogeneous Feedback, (Offline Safe POHF)
• lemma 1
• lemma 2: Feasibility generalization bound
• lemma 3
• lemma 4: Feasibility generalization bound