Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models

TL;DR

MoP-RL tackles preference-based reinforcement learning when system dynamics are unknown by jointly learning a dynamics model and a reward predictor, then planning with a model-based controller. It introduces reward pre-training from suboptimal demonstrations, and active, information-gain driven generation of human preferences from trajectories produced during planning. The approach demonstrates improved sample efficiency and safety over model-free PbRL across four robotics-inspired domains, including high-dimensional visual tasks and a Hopper backflip. By reusing the learned dynamics for multiple users and enabling environment-free preference queries, MoP-RL offers practical benefits for personalized, safe robot learning.

Abstract

Preference-based reinforcement learning (PbRL) can enable robots to learn to perform tasks based on an individual's preferences without requiring a hand-crafted reward function. However, existing approaches either assume access to a high-fidelity simulator or analytic model or take a model-free approach that requires extensive, possibly unsafe online environment interactions. In this paper, we study the benefits and challenges of using a learned dynamics model when performing PbRL. In particular, we provide evidence that a learned dynamics model offers the following benefits when performing PbRL: (1) preference elicitation and policy optimization require significantly fewer environment interactions than model-free PbRL, (2) diverse preference queries can be synthesized safely and efficiently as a byproduct of standard model-based RL, and (3) reward pre-training based on suboptimal demonstrations can be performed without any environmental interaction. Our paper provides empirical evidence that learned dynamics models enable robots to learn customized policies based on user preferences in ways that are safer and more sample efficient than prior preference learning approaches. Supplementary materials and code are available at https://sites.google.com/berkeley.edu/mop-rl.

Figures (4)

• Figure 1: Model-Based Preference-Based RL (MoP-RL): We leverage intrinsic exploration (left) to learn a dynamics model that enables efficient reward pre-training (center) by running behavioral cloning (BC) on a small number of demonstrations and using noise injection to simulate a variety of worse trajectories using rollouts in the learned dynamics model. We fine-tune the learned reward function via active preference queries to a human (right), where the trajectories for queries are simulated using the learned dynamics model and produced as a byproduct of running model-based RL with the learned reward function. MoP-RL uses the learned dynamics model to enable safe and sample efficient learning of human preferences.
• Figure 2: Cross Entropy Method: For an initial state $s_0$, we sample a set of action sequences and predict the corresponding state trajectory for each one using the dynamics model $\hat{f}_{\phi}$. We estimate rewards for these trajectories as a linear combination of the reward model $\hat{r}_{\theta}$'s prediction and an RND bonus. The $m_e$ action sequences with the highest predicted rewards form the elites; these are used to update the CEM distribution and sample the next population of action sequences. In the final CEM iteration, the mean of the action distribution of elites is selected for execution in the environment.
• Figure 3: Assistive Gym start (left) and goal (right) state.
• Figure 4: MoP-RL trains a Hopper to perform a backflip via preference queries over learned dynamics.