Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations

TL;DR

This work addresses cross-embodiment inverse reinforcement learning from mixed-quality video demonstrations (MQME) to learn embodiment-agnostic rewards. It analyzes several human-feedback–driven approaches—X-RLHF, representation-learning-from-preferences (XPrefs, XTriplets), and XIRL-Buckets—alongside an XIRL baseline to understand how mixed quality affects transfer. Experiments on the X-MAGICAL domain show that X-RLHF and XIRL-Buckets achieve robust cross-embodiment transfer despite MQME data, while XIRL degrades and XTriplets underperform, underscoring the value of representation alignment guided by human feedback. The results highlight the importance of alignment through preferences for transferable representations, while also indicating remaining gaps compared to near-optimal demonstrations and opportunities for more label-efficient approaches.

Abstract

We study the problem of cross-embodiment inverse reinforcement learning, where we wish to learn a reward function from video demonstrations in one or more embodiments and then transfer the learned reward to a different embodiment (e.g., different action space, dynamics, size, shape, etc.). Learning reward functions that transfer across embodiments is important in settings such as teaching a robot a policy via human video demonstrations or teaching a robot to imitate a policy from another robot with a different embodiment. However, prior work has only focused on cases where near-optimal demonstrations are available, which is often difficult to ensure. By contrast, we study the setting of cross-embodiment reward learning from mixed-quality demonstrations. We demonstrate that prior work struggles to learn generalizable reward representations when learning from mixed-quality data. We then analyze several techniques that leverage human feedback for representation learning and alignment to enable effective cross-embodiment learning. Our results give insight into how different representation learning techniques lead to qualitatively different reward shaping behaviors and the importance of human feedback when learning from mixed-quality, mixed-embodiment data.

Figures (4)

• Figure 1: Policy evaluations. Shading denotes standard error bars. Compared with the Ground Truth reward and XIRL (optimal demonstrations) oracles, we find that XIRL using mixed quality data suffers a significant performance drop, while X-RLHF and XIRL-Buckets perform much better. Simply training a goal classifier is insufficient and pretraining a representation based on triplet queries also fails to perform well.
• Figure 2: Qualitative Analysis of Learned Rewards and Representations. We compare the true reward with the predicted rewards across a failed (a) and successful (b) trajectory.
• Figure 3: XPrefs Performance.(a) Static vs. Dynamic Goal Embeddings for XPrefs. We find that using dynamic goal embeddings that were periodically updated every X steps during training leads to training instabilities, hindering good representation learning. By contrast, using a static goal embedding leads to a convergent training loss. (b) The RL performance of XPrefs and XPrefs with the goal representation ($g$) hard-coded as the origin. In both cases, XPrefs obtains very similar results to X-RLHF
• Figure 4: Correlation Plots. The correlation between ground truth reward ($r$) and learned reward ($\hat{r}$, normalized to range of $r$) for the reward learning test demonstration set. Rewards shown are cumulative rewards over full trajectories. Reward learning trains on the XMagical gripper, longstick, and shortstick embodiments (Columns 1-3) and then is tested on the withheld mediumstick embodiment (Column 4).