XIRL: Cross-embodiment Inverse Reinforcement Learning

TL;DR

XIRL tackles the challenge of learning from third-person demonstrations across different embodiments by learning embodiment-invariant visual embeddings using temporal cycle-consistency (TCC). It then defines a dense, embedding-based reward r(s) that encodes task progress and can be used with RL (SAC) to learn policies for unseen embodiments. The approach is validated on the X-MAGICAL benchmark and a real-world cross-embodiment dataset (X-REAL), showing improved sample efficiency and generalization over baselines. Overall, XIRL enables scalable, label-free reward learning from videos and demonstrates robust cross-embodiment transfer without requiring aligned frame correspondences.

Abstract

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings typically required alignment with a reference trajectory, which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings are aware of task progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. Additionally, when transferring real-world human demonstrations to a simulated robot, we find that XIRL is more sample efficient than current best methods. Qualitative results, code, and datasets are available at https://x-irl.github.io

Figures (16)

• Figure 1: Cross-embodiment Inverse Reinforcement Learning. We learn embodiment-invariant visual representations from offline video demonstrations (stick agents on the left) using TCC Dwibedi_2019_CVPR, then use the trained encoder to generate embodiment-invariant visual reward functions that can be used to learn policies on new embodiments (gripper on the right) with reinforcement learning.
• Figure 2: Difference in state visitation distributions across different embodiments performing the same task in X-MAGICAL (Section \ref{['subsection:x-magical']}).
• Figure 3: Same-embodiment setting: Comparison of XIRL with other baseline reward functions, using SAC haarnoja2018soft for RL policy learning on the X-MAGICAL sweeping task.
• Figure 4: Cross-embodiment setting: XIRL performs favorably when compared with other baseline reward functions, trained on observation-only demonstrations from different embodiments. Each agent {long-stick, medium-stick, short-stick, gripper} is shown using demonstrations from the other 3 embodiments, with SAC haarnoja2018soft for RL policy learning on the X-MAGICAL sweeping task.
• Figure 5: Real-world-demo cross-embodiment setting: Comparison of XIRL with baselines using the simulated State Pusher environment from schmeckpeper2020reinforcement. XIRL (real only) leverages real-world demonstration videos of humans (left, row 2) to teach a robot arm in sim (left, row 1), but unlike schmeckpeper2020reinforcement, we do not use human-labeled data of paired frame correspondences. RLV* denotes results taken verbatim from schmeckpeper2020reinforcement which uses a different implementation of SAC for RL policy learning.