Learning Human Objectives by Evaluating Hypothetical Behavior

TL;DR

This work tackles aligning RL agents to user objectives when the dynamics and reward structure are unknown by learning a user reward model from labels on hypothetical, synthesized trajectories. The core method, ReQueST, builds a generative trajectory model and represents rewards as an ensemble classifier over {good, unsafe, neutral}, guided by four acquisition functions that promote informativeness, safety, and diversity. By combining trajectory optimization with model-based RL (MPC) and safety-focused data collection, ReQueST achieves robust transfer to new initial-state distributions and demonstrates reward-hacking detection and safe exploration. The approach reduces user labeling requirements and shows strong performance across MNIST, a 2D navigation task, and Car Racing, highlighting practical impact for deploying safe, goal-aligned agents in complex environments.

Abstract

We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and interactively learns a model of the user's reward function. We start with a generative model of initial states and a forward dynamics model trained on off-policy data. Our method uses these models to synthesize hypothetical behaviors, asks the user to label the behaviors with rewards, and trains a neural network to predict the rewards. The key idea is to actively synthesize the hypothetical behaviors from scratch by maximizing tractable proxies for the value of information, without interacting with the environment. We call this method reward query synthesis via trajectory optimization (ReQueST). We evaluate ReQueST with simulated users on a state-based 2D navigation task and the image-based Car Racing video game. The results show that ReQueST significantly outperforms prior methods in learning reward models that transfer to new environments with different initial state distributions. Moreover, ReQueST safely trains the reward model to detect unsafe states, and corrects reward hacking before deploying the agent.

Figures (14)

• Figure 1: Our method learns a reward model from user feedback on hypothetical behaviors, then deploys a model-based reinforcement learning agent that optimizes the learned rewards.
• Figure 2: Our method automatically synthesizes hypotheticals like the trolley problem. Consider a training environment in which the following two states are common: either one of the tracks is empty, or there are fewer people on the current track than the other track. In these states, the consequentialist and deontologist reward functions agree. After asking the user to label these states, we are not able to determine which of the two is the true reward function, since both are consistent with the training data. Our method queries the user for labels at states where the value of information is highest: states where there are more people on the current track than the other track, but there are still some people on the other track. By eliciting user labels at these unlikely-but-informative states, we learn a reward model that more accurately captures the user's objectives.
• Figure 3: Left: The 2D navigation task, where the agent navigates to the goal region (green) in the lower left while avoiding the trap region (red) in the upper right. The agent starts in the lower left corner in the training environment, and starts in the upper right corner in the test environment. Right: Examples of hypothetical states synthesized throughout learning, illustrating the qualitative differences in the behaviors targeted by each AF.
• Figure 4: Experiments that address Q1 -- does synthesizing hypothetical trajectories elicit more informative labels than rolling out a policy in the training environment? -- by comparing our method, which uses synthetic trajectories, to baselines that only use real trajectories generated in the training environment. The results on MNIST, 2D navigation, and Car Racing show that our method (orange) significantly outperforms the baselines (blue and gray), which never succeed in 2D navigation. The x-axis represents the number of queries to the user, where each query elicits a label for a single state transition $(s, a, s')$. The shaded areas show standard error over three random seeds.
• Figure 5: Experiments that address Q2 -- can our method detect and correct reward hacking? -- by comparing our method, which uses synthetic trajectories, to baselines that only use real trajectories generated in the training environment. The results on 2D navigation show that our method (orange) significantly outperforms the baselines (blue and gray). The x-axis represents the number of queries to the user, where each query elicits a label for a single state transition $(s, a, s')$. The shaded areas show standard error over three random seeds.