Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment

TL;DR

RAPL introduces an observation-only framework to align visuomotor policies with end-user preferences by first aligning the robot’s visual representation to the human’s using limited preference data, then constructing a dense visual reward via optimal transport in the aligned space. The method demonstrates strong data efficiency and cross-embodiment generalization in both simulated benchmarks and real-world diffusion-policy alignment, requiring up to 5x less real human feedback than traditional RLHF. Empirical results show RAPL-rewarded policies closely track ground-truth rewards, outperform baselines on several tasks, and transfer across embodiments without retraining the reward model. This approach offers a scalable path to user-aligned visuomotor robotics by leveraging representation alignment and OT-based reward construction. All mathematical notions are presented with $...$ delimiters to facilitate precise downstream processing.

Abstract

Visuomotor robot policies, increasingly pre-trained on large-scale datasets, promise significant advancements across robotics domains. However, aligning these policies with end-user preferences remains a challenge, particularly when the preferences are hard to specify. While reinforcement learning from human feedback (RLHF) has become the predominant mechanism for alignment in non-embodied domains like large language models, it has not seen the same success in aligning visuomotor policies due to the prohibitive amount of human feedback required to learn visual reward functions. To address this limitation, we propose Representation-Aligned Preference-based Learning (RAPL), an observation-only method for learning visual rewards from significantly less human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback on fine-tuning pre-trained vision encoders to align with the end-user's visual representation and then constructs a dense visual reward via feature matching in this aligned representation space. We first validate RAPL through simulation experiments in the X-Magical benchmark and Franka Panda robotic manipulation, demonstrating that it can learn rewards aligned with human preferences, more efficiently uses preference data, and generalizes across robot embodiments. Finally, our hardware experiments align pre-trained Diffusion Policies for three object manipulation tasks. We find that RAPL can fine-tune these policies with 5x less real human preference data, taking the first step towards minimizing human feedback while maximizing visuomotor robot policy alignment.

Figures (11)

• Figure 1: Representation-Aligned Preference-based Learning (RAPL), is an observation-only method for learning visual robot rewards from significantly less human preference feedback. (center) Unlike traditional reinforcement learning from human feedback, RAPL focuses human feedback on fine-tuning pre-trained vision encoders to align with the end-user’s visual representation. The aligned representation is used to construct an optimal transport-based visual reward for aligning the robot's visuomotor policy. (left) Before alignment, the robot frequently picks up a bag of chips by squeezing the middle, risking damage to the contents. (right) After alignment with our RAPL reward, the robot adheres to the end-user's preference and picks up the bag by its edges.
• Figure 2: X-Magical & IsaacGym tasks. Top row are high-reward behaviors and bottom row are low-reward behaviors according to the human's preferences.
• Figure 3: X-Magical. (left & right) examples of preferred and disliked videos for each task. (center) reward associated with each video under each method. RAPL's predicted reward follows the GT pattern: low reward when the behavior are disliked and high reward when the behavior are preferred. RLHF and TCC assign high reward to disliked behavior (e.g., (D)).
• Figure 4: X-Magical. Policy evaluation success rate during policy learning. Colored lines are the mean and variance of the evaluation success rate. RAPL can match GT in the avoiding task and outperforms baseline visual rewards in grouping task.
• Figure 5: Manipulation: Reward Prediction. (center) Expert, preferred, and disliked video demo. (left) Reward of each video under each method. RAPL's predicted reward follows the GT pattern. RLHF assigns high reward to disliked behavior. (right) OT coupling matrix for each visual representation. Columns are embedded frames of expert demo. Rows of top matrices are embedded frames of preferred demo; rows of bottom matrices are embedded frames of disliked demo. Peaks exactly along the diagonal indicate that the frames of the two videos are aligned in the latent space; uniform values in the matrix indicate that the two videos cannot be aligned (i.e., all frames are equally "similar’’ to the next). RAPL exhibits the diagonal peaks for expert-and-preferred and uniform for expert-and-disliked, while baselines show diffused values no matter the videos being compared.

Theorems & Definitions (2)

• Definition 1: Triplet-based Representation Space
• Definition 2: Visual Representation Alignment Problem