PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation

TL;DR

PEARL addresses the data-efficiency challenge of preference-based RL by introducing Cross-task Preference Alignment (CPA), which uses Gromov-Wasserstein distance to align source and target trajectories and transfer source preferences without target labels. It couples CPA with Robust Reward Learning (RRL), modeling rewards as Gaussian distributions to capture uncertainty and improve robustness to noisy transferred labels. Empirical results on Meta-World and Robomimic show that CPA-derived labels enable competitive zero-shot and few-shot performance, with RPT+CPA approaching oracle baselines and outperforming several baselines under label scarcity. The approach substantially reduces human labeling requirements while maintaining strong policy quality, offering practical benefits for real-world robotic manipulation tasks.

Abstract

In preference-based Reinforcement Learning (RL), obtaining a large number of preference labels are both time-consuming and costly. Furthermore, the queried human preferences cannot be utilized for the new tasks. In this paper, we propose Zero-shot Cross-task Preference Alignment and Robust Reward Learning (PEARL), which learns policies from cross-task preference transfer without any human labels of the target task. Our contributions include two novel components that facilitate the transfer and learning process. The first is Cross-task Preference Alignment (CPA), which transfers the preferences between tasks via optimal transport. The key idea of CPA is to use Gromov-Wasserstein distance to align the trajectories between tasks, and the solved optimal transport matrix serves as the correspondence between trajectories. The target task preferences are computed as the weighted sum of source task preference labels with the correspondence as weights. Moreover, to ensure robust learning from these transferred labels, we introduce Robust Reward Learning (RRL), which considers both reward mean and uncertainty by modeling rewards as Gaussian distributions. Empirical results on robotic manipulation tasks from Meta-World and Robomimic demonstrate that our method is capable of transferring preference labels across tasks accurately and then learns well-behaved policies. Notably, our approach significantly exceeds existing methods when there are few human preferences. The code and videos of our method are available at: https://sites.google.com/view/pearl-preference.

Figures (5)

• Figure 1: Framework of PEARL. Given unlabeled target task trajectories and source task trajectories and their preference labels, the trajectories between tasks are first aligned via Gromov-Wasserstein distance. Then the target task preference labels are computed by the solved optimal transport matrix and source task preference labels. The reward model is learned robustly and finally offline RL algorithm is applied to obtain the policy.
• Figure 2: Diagram of cross-task preference alignment. The circle $\bigcirc$ represents a trajectory segment in each task. (a) CPA uses Gromov-Wasserstein distance as a relational distance metric to align trajectory distributions between source and target tasks. (b) The optimal transport matrix is solved by optimal transport, with each element representing the correspondence between trajectories of two tasks. (c) The preference labels of trajectory pairs of the target task are computed based on trajectory correspondence by Equation \ref{['eq:CPA']}. $z(y_1,y_3)=1$ indicates that $y_3$ is better than $y_1$ and $0$ indicates $y_1$ is preferred.
• Figure 3: Different types of reward modeling. (a) Scalar reward modeling, which only considers scalar rewards. This modeling type is widely used in preference-based RL algorithms christiano2017deeplee2021pebblekim2023preference. (b) Distributional reward modeling, which adds a branch for modeling reward uncertainty in addition to reward mean.
• Figure 4: Success rate of Door Close, Window Open and Lift-MH with different scripted preference labels.
• Figure 5: Success rate of Sweep Into, Window Open and Lift-MH under different noise levels.

Theorems & Definitions (1)

• Definition 3.1