Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input

TL;DR

This work introduces Pragmatic Feature Preferences, a framework that learns reward models from human input by combining feature-level preferences with traditional example-level comparisons and linguistic descriptions. By coupling feature-level feedback with pragmatic data augmentation, the method enables faster and more data-efficient learning in contextual bandits, demonstrated in both vision (mushroom foraging) and language (flight booking) domains, and validated by a real-user study showing comparable effort to baseline queries. Key contributions include a formal joint loss that blends RLHF and feature-pairwise signals, a pragmatic augmentation strategy to synthesize additional labeled data, and empirical evidence that pruning away non-reward-relevant features accelerates convergence, especially under feature sparsity. The approach promises more efficient, user-aligned reward learning and highlights the value of modeling human pragmatic teaching signals in AI systems.

Abstract

Humans use social context to specify preferences over behaviors, i.e. their reward functions. Yet, algorithms for inferring reward models from preference data do not take this social learning view into account. Inspired by pragmatic human communication, we study how to extract fine-grained data regarding why an example is preferred that is useful for learning more accurate reward models. We propose to enrich binary preference queries to ask both (1) which features of a given example are preferable in addition to (2) comparisons between examples themselves. We derive an approach for learning from these feature-level preferences, both for cases where users specify which features are reward-relevant, and when users do not. We evaluate our approach on linear bandit settings in both vision- and language-based domains. Results support the efficiency of our approach in quickly converging to accurate rewards with fewer comparisons vs. example-only labels. Finally, we validate the real-world applicability with a behavioral experiment on a mushroom foraging task. Our findings suggest that incorporating pragmatic feature preferences is a promising approach for more efficient user-aligned reward learning.

Figures (6)

• Figure 1: A. An illustrative user reward function in the mushroom foraging task. Rewards are a linear combination of color, shape, and weight features. B.Example preference queries learn a traditional RLHF loss over example-level comparisons. C. Our approach, pragmatic feature preference queries, makes use of (1) fine-grained feature-level preferences in conjunction with example-level preferences, and (2) language descriptions to infer reward-relevant features and augment preference data.
• Figure 2: Results with simulated preference labels on the mushroom foraging task. Prag FP outperforms other methods, converging to a more accurate learned reward given fewer seen examples. This effect is especially prominent as the reward-relevant features become more sparse. Confidence bounds depict standard error across five independent seeds.
• Figure 3: Results with real user descriptions on the flight booking task across 20 randomly sampled reward functions. Confidence bounds depict standard error across five independent seeds.
• Figure 4: Results with real user responses on the mushroom foraging task. These results corroborate the simulated results from \ref{['fig:sim_mushrooms']}. Confidence bounds depict standard error across five independent seeds.
• Figure 5: Example reward function provided in the user study.