What Matters to You? Towards Visual Representation Alignment for Robot Learning

TL;DR

This work tackles misalignment between robot rewards and human preferences when rewards derive from visual inputs. It introduces Representation-Aligned Preference-based Learning (RAPL), which uses human video preferences to align the robot’s visual representation with the human’s and then employs optimal transport to define a visually grounded reward that favors human-aligned behavior. Across X-MAGICAL and robotic manipulation tasks, RAPL achieves high task success with superior sample efficiency and exhibits strong zero-shot generalization when the visual encoder is learned on a different embodiment. The approach eliminates reliance on action labels or proxy tasks for representation learning, enabling end-user-centric reward learning directly from video preferences and embedding distributions.

Abstract

When operating in service of people, robots need to optimize rewards aligned with end-user preferences. Since robots will rely on raw perceptual inputs like RGB images, their rewards will inevitably use visual representations. Recently there has been excitement in using representations from pre-trained visual models, but key to making these work in robotics is fine-tuning, which is typically done via proxy tasks like dynamics prediction or enforcing temporal cycle-consistency. However, all these proxy tasks bypass the human's input on what matters to them, exacerbating spurious correlations and ultimately leading to robot behaviors that are misaligned with user preferences. In this work, we propose that robots should leverage human feedback to align their visual representations with the end-user and disentangle what matters for the task. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem and visual reward learning problem through the lens of preference-based learning and optimal transport. Across experiments in X-MAGICAL and in robotic manipulation, we find that RAPL's reward consistently generates preferred robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation is learned from a different embodiment than the robot's.

Figures (17)

• Figure 1: Representation-Aligned Preference-based Learning (RAPL), is an action-free visual representation learning method using easy-to-provide human preference feedback on video demos. Using the human preference triplets, the robot goes from paying attention to the end-effector ($\tilde{\phi}_{\mathrm{H}}^0$) to paying attention to the objects and the goal region ($\tilde{\phi}_{\mathrm{H}}^*$) at the end of alignment. The aligned representation is used to construct an optimal transport-based visual reward for robot behavior learning.
• Figure 2: X-Magical & IsaacGym tasks.
• Figure 3: X-Magical. (left & right) examples of preferred and disliked videos for each task. (center) reward associated with each video under each method. RAPL's predicted reward follows the GT pattern: low reward when the behavior are disliked and high reward when the behavior are preferred. RLHF and TCC assign high reward to disliked behavior (e.g., (D)).
• Figure 4: X-Magical. Policy evaluation success rate during policy learning. Colored lines are the mean and variance of the evaluation success rate. RAPL can match GT in the avoiding task and outperforms baseline visual rewards in grouping task.
• Figure 5: Manipulation. (center) Expert, preferred, and disliked video demo. (left) reward associated with each video under each method. RAPL's predicted reward follows the GT pattern. RLHF assigns high reward to disliked behavior. (right) OT coupling for each representation. Columns are embedded frames of expert demo. Rows of top matrices are embedded frames of preferred demo; rows of bottom matrices are embedded frames of disliked demo. Peaks exactly along the diagonal indicate that the frames of the two videos are aligned in the latent space; uniform values in the matrix indicate that the two videos cannot be aligned (i.e., all frames are equally "similar’’ to the next). RAPL matches this structure: diagonal peaks for expert-and-preferred and uniform for expert-and-disliked, while baselines show diffused values no matter the videos being compared.

Theorems & Definitions (2)

• Definition 1: Triplet-based Representation Space
• Definition 2: Visual Representation Alignment Problem