Learning Contextually-Adaptive Rewards via Calibrated Features

TL;DR

This work tackles context-dependent reward learning by explicitly modeling how context modulates the saliency of reward features rather than the underlying preferences. It introduces calibrated features—context-conditioned feature mappings—and learns them through contextual feature queries grounded in Bradley–Terry likelihood, separating context effects from context-invariant rewards. Empirical results in simulation show that calibrating features yields higher reward accuracy with an order of magnitude fewer preference queries and better low-data performance, while a human user study confirms feasibility and personalization of contextual preferences. The approach supports modular, reusable representations that can be composed to form context-adaptive rewards, with potential for improved interpretability and practical deployment in personalizable robotic systems.

Abstract

A key challenge in reward learning from human input is that desired agent behavior often changes based on context. For example, a robot must adapt to avoid a stove once it becomes hot. We observe that while high-level preferences (e.g., prioritizing safety over efficiency) often remain constant, context alters the $\textit{saliency}$--or importance--of reward features. For instance, stove heat changes the relevance of the robot's proximity, not the underlying preference for safety. Moreover, these contextual effects recur across tasks, motivating the need for transferable representations to encode them. Existing multi-task and meta-learning methods simultaneously learn representations and task preferences, at best $\textit{implicitly}$ capturing contextual effects and requiring substantial data to separate them from task-specific preferences. Instead, we propose $\textit{explicitly}$ modeling and learning context-dependent feature saliency separately from context-invariant preferences. We introduce $\textit{calibrated features}$--modular representations that capture contextual effects on feature saliency--and present specialized paired comparison queries that isolate saliency from preference for efficient learning. Simulated experiments show our method improves sample efficiency, requiring 10x fewer preference queries than baselines to achieve equivalent reward accuracy, with up to 15% better performance in low-data regimes (5-10 queries). An in-person user study (N=12) demonstrates that participants can effectively teach their personal contextual preferences with our method, enabling adaptable and personalized reward learning.

Figures (19)

• Figure 1: Left: three manipulation environments. Right: calibrated features learned from simulated human input. Learned contextual effects reshape the base feature as the relevant contextual element changes. Point clouds show 8k EE locations, normalized over the full range of the relevant contextual element. Irrelevant contextual elements fixed to zero.
• Figure 2: Exp. 1 & 2 reward accuracy results. Results shown for each environment (rows) when learning rewards that capture contextual effects on one of the three environment features (columns 1-3) or all three environment features (column 4).
• Figure 3: Exp 1 & 2 induced trajectory reward results, showing ground truth reward of trajectories selected by learned rewards. Results shown for each environment for rewards in which a single or all three environment features are contextually-affected.
• Figure 4: Subjective user ratings and model ranking improved with increasing training queries (*p<0.05). The human feature benefits from 100 vs. 50 queries, while the stove feature plateaus after 50 queries, highlighting that query requirements vary with feature complexity. Boxes show 25th–75th percentiles and median, whiskers mark 1.5$\times$IQR, and points mark outliers.
• Figure 5: Examples of two unique user contextual preferences for how utensil sharpness should affect the human feature (EE xy-planar distance to human). Users differ in their comfort with robot proximity and in which utensils they view as similar. By labeling contextual feature queries, users trained calibrated features that accurately captured these preferences.