A Generalized Acquisition Function for Preference-based Reward Learning

TL;DR

The paper introduces a generalized acquisition function for preference-based reward learning that optimizes reward alignment up to behavioral equivalence rather than exact parameter recovery. It defines flexible alignment metrics and a tractable proxy objective, solved greedily to select informative queries that improve downstream behavior under the chosen metric. The authors prove that, for adaptive monotone and adaptive submodular objectives, the greedy policy achieves near-optimal improvement, and demonstrate empirically that the method outperforms mutual information baselines across synthetic, assistive robotics, and NLP tasks, including successful domain transfer. This approach reduces data requirements in user feedback and enables task-relevant reward learning in deployment settings with distribution shifts. The framework accommodates linear and nonlinear rewards and opens avenues for diverse alignment metrics and gradient-based extensions.

Abstract

Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task. Previous works have shown that actively synthesizing preference queries to maximize information gain about the reward function parameters improves data efficiency. The information gain criterion focuses on precisely identifying all parameters of the reward function. This can potentially be wasteful as many parameters may result in the same reward, and many rewards may result in the same behavior in the downstream tasks. Instead, we show that it is possible to optimize for learning the reward function up to a behavioral equivalence class, such as inducing the same ranking over behaviors, distribution over choices, or other related definitions of what makes two rewards similar. We introduce a tractable framework that can capture such definitions of similarity. Our experiments in a synthetic environment, an assistive robotics environment with domain transfer, and a natural language processing problem with real datasets demonstrate the superior performance of our querying method over the state-of-the-art information gain method.

Figures (3)

• Figure 1: Results of the synthetic environment experiment over 50 seeds (mean$\pm$se).
• Figure 2: Results of the Assistive Gym experiment over 100 seeds (mean$\pm$se).
• Figure 3: NLP experiments results over 50 seeds (mean$\pm$se).

Theorems & Definitions (4)

• Remark 1
• proof
• Remark 2
• proof