Pre-training Auto-regressive Robotic Models with 4D Representations

TL;DR

ARM4R presents an Auto-regressive Robotic Model that learns 4D representations by lifting 2D video observations to 3D point tracks across time, enabling transfer from abundant human videos to robotic control. It trains in three stages—large-scale 3D point tracking on Epic-Kitchens100, robotic fine-tuning, and robotic control—achieving strong performance on RLBench tasks and real-robot experiments with cross-robot generalization. Empirical results show ARM4R outperforming state-of-the-art baselines and highlight the value of 4D pre-training over robot-only approaches. The work emphasizes data-efficient robotics by leveraging human video data and 4D representations to bridge the domain gap between humans and robots.

Abstract

Foundation models pre-trained on massive unlabeled datasets have revolutionized natural language and computer vision, exhibiting remarkable generalization capabilities, thus highlighting the importance of pre-training. Yet, efforts in robotics have struggled to achieve similar success, limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world. In this paper, we introduce ARM4R, an Auto-regressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre-trained robotic model. Specifically, we focus on utilizing 3D point tracking representations from videos derived by lifting 2D representations into 3D space via monocular depth estimation across time. These 4D representations maintain a shared geometric structure between the points and robot state representations up to a linear transformation, enabling efficient transfer learning from human video data to low-level robotic control. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.

Figures (8)

• Figure 1: Overview of ARM4R. We introduce an Auto-regressive Robotic Model that leverages low-level 4D Representations (3D point tracks across time) learned from human videos to yield a better pre-trained robotic model.
• Figure 2: ARM4R is trained in three stages. Top Grey Box: The first two stages focus on learning a scene-wide 4D representation by predicting 3D points across time, where Stage 1 pre-trains on a large egocentric human dataset (Epic-Kitchens100), and Stage 2 fine-tunes on a smaller dataset (1-2K demonstrations) of robotic scenes, adapting the point tracking to robotic scene and camera. Bottom Grey Box: Finally, the model is fine-tuned to predict robot proprioceptive states rather than 3D points to enable robotic control.
• Figure 3: Ablation Study for Stages 1 and 2. We train ARM4R on three real tasks in the Kinova setting, ablating Stages 1 and 2. The results indicate that while both stages improve performance, Stage 1 has a more significant impact.
• Figure 4: Visualization of ARM4R's 3D Point Track results on randomly chosen Epic-Kitchens (in-domain) and Ego-4D (out-of-domain) human videos.
• Figure 5: Visualization of ARM4R's 3D Point Track results on randomly chosen Kinova (in-domain) and Open X-Embodiment (out-of-domain) robot videos.