Offline Preference-Based Apprenticeship Learning

TL;DR

Offline Preference-Based Apprenticeship Learning (OPAL) enables learning user-intended behaviors without environment interaction by extracting informative trajectory snippets from offline data, learning a distribution over reward functions from human preferences, and then applying offline RL to customize the policy. The approach combines two offline stages: reward learning via a Bradley-Terry model with uncertainty representations (ensembles or Bayesian dropout) and active query selection (disagreement or information gain) to efficiently elicit preferences. Empirical results on D4RL benchmarks reveal that many tasks degrade when rewards are uninformative, validating reward learning in offline settings, and show that ensemble-based disagreement often yields the best active queries, enabling near-ground-truth performance with relatively few queries. The paper also introduces new offline tasks to stress open-ended reward learning, demonstrating that diverse offline data can support flexible, user-specific behaviors while maintaining safety and efficiency in learning.

Abstract

Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning, learns a distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we identify a subset of existing offline RL benchmarks that are well suited for offline reward learning and also propose new offline apprenticeship learning benchmarks which allow for more open-ended behaviors. Our empirical results suggest that combining offline RL with learned human preferences can enable an agent to learn to perform novel tasks that were not explicitly shown in the offline data.

Figures (7)

• Figure 1: Offline Preference-Based Apprenticeship Learning (OPAL) enables safe and efficient preference-based policy customization without any environmental interactions. Given an offline database consisting of trajectories, OPAL queries an expert for preference labels over trajectory segments from the database, learns a reward function from preferences, and then performs offline RL using rewards provided by the learned reward function.
• Figure 2: OPAL selects active queries from an offline dataset consisting of random point to point navigation. These active queries elicit preferences about a human's preferences (the human wants the robot to perform counter clockwise orbits and prefers blue over red trajectories). The learned reward function matches the human's preferences and, when combined with offline RL, leads to an appropriate policy that looks significantly different from the original offline data distribution. Offline data is often not collected for the specific task you want, but for other tasks. Thus, being able to repurpose data from a variety of sources is important for generalizing to different user preferences in offline settings where we can’t easily just gather lots of new online data.
• Figure 3: Experimental domains chosen from D4RL fu2020d4rl for use in offline preference-based apprenticeship learning.
• Figure 4: (a) Ensemble disagreement after 10 rounds has similar performance to ground truth reward in Maze2d-Umaze. (b) Ensemble disagreement after 10 rounds has slightly worse performance compared to ground truth reward in Maze2d-Medium. (c) Ensemble disagreement after 10 rounds has similar performance compared to ground truth reward in Halfcheetah.
• Figure 5: Constrained Goal Navigation. The highlighted yellow region represents a constraint region that the human prefers the agent to avoid while also traveling to the goal position shown in red. OPAL produces trajectories that match this preference by taking a more round-about, but more preferred, path to the goal.