Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

TL;DR

A class of multimodal RLHF methods based on a latent variable formulation that enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Figures (13)

• Figure 1: Current RLHF approaches ouyang2022training incorrectly assume a unimodal BTL reward model for a diverse population of users. In this example, users have diverging preferences over the level of detail provided in the responses from a large language model. Without additional context, the BTL model considers both responses to be equally likely. In contrast, our method, VPL, is a personalized approach to RLHF. Using a few samples from a particular user, we infer the distribution over their distinct preferences. Based on this distribution, we condition the reward model to more accurately predict rewards, and enable steering the resulting policy to personalize to the specific user. This enables accounting for and serving the preferences of under-represented groups which would otherwise be ignored by the standard BTL model ouyang2022training.
• Figure 2: VPL LLM architecture for reward learning. The left and right parts denote the encoder $q_\psi$ and the reward model $r(s, z)$ respectively.
• Figure 3: Ground truth preferences (a) show that annotators prefer the robot navigate to two different goals. Unimodal BTL (b) averages over the two modes. VPL (c) accurately reconstructs diverse preferences, and learns $z$ conditioned policies that can reach either goal.
• Figure 4: Performance of a downstream policy on diverse control and reasoning tasks, using the rewards trained using different baselines. We report the mean and standard error over five seeds. Note: Habitat envs have a one-step greedy policy so reward scaling and SPO+VPL are not required.
• Figure 5: Active learning enables personalizing policies to user preferences with fewer queries.