Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

Yifeng Zhu, Zhenyu Jiang, Peter Stone, Yuke Zhu

TL;DR

GROOT addresses the challenge of generalizing imitation-learned manipulation policies to perceptual variations by constructing object-centric 3D representations and processing them with a transformer-based policy. It integrates interactive scribble-based object annotation, VOS-driven tracking, 3D backprojection, and a segmentation-correspondence module using open-vocabulary and semantic-feature models to enable testing with unseen objects. The approach demonstrates strong generalization to background changes, camera viewpoint shifts, and new object instances in both simulated and real robot tasks, outperforming prior end-to-end and object-proposal methods. This work advances practical robot manipulation by enabling robust, generalizable policies trained from demonstrations in a single environment.

Abstract

We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup. More videos and model details can be found in the appendix and the project website: https://ut-austin-rpl.github.io/GROOT .

Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

TL;DR

GROOT addresses the challenge of generalizing imitation-learned manipulation policies to perceptual variations by constructing object-centric 3D representations and processing them with a transformer-based policy. It integrates interactive scribble-based object annotation, VOS-driven tracking, 3D backprojection, and a segmentation-correspondence module using open-vocabulary and semantic-feature models to enable testing with unseen objects. The approach demonstrates strong generalization to background changes, camera viewpoint shifts, and new object instances in both simulated and real robot tasks, outperforming prior end-to-end and object-proposal methods. This work advances practical robot manipulation by enabling robust, generalizable policies trained from demonstrations in a single environment.

Abstract

We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup. More videos and model details can be found in the appendix and the project website: https://ut-austin-rpl.github.io/GROOT .
Paper Structure (42 sections, 1 equation, 7 figures, 3 tables)

This paper contains 42 sections, 1 equation, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Imitation Learning for Robot Manipulation
  4. Representations Learning for Vision-Based Manipulation
  5. Object-Centric Representation Learning
  6. GROOT
  7. Problem Formulation
  8. Dimensions of Generalization
  9. Building Object-Centric 3D Representations
  10. Scribble Annotation of Task-Relevant Objects
  11. Tracking Task-Relevant Objects Over Time
  12. Backprojection to Point Clouds
  13. Policy Design
  14. Segmentation Correspondence Model
  15. Experiments
  16. ...and 27 more sections

Figures (7)

  • Figure 1: GROOT Overview.GROOT learns closed-loop visuomotor policies from demonstrations under a single setup, and generalizes to setups with different conditions, namely different visual distractions, changed camera angles, and new objects.
  • Figure 2: GROOT Model Architecture.GROOT leverages an interactive segmentation model, S2M, to obtain a single-frame annotation from demonstrators. Then a Video Object Segmentation model, XMem, propagates segmentation masks across time frames. The object masks are then back-projected into point clouds, and a transformer-based policy processes the point clouds to output actions. During deployment, GROOT uses a segmentation correspondence model based on an open-vocabulary segmentation model (SAM) and a pretrained semantic feature model (DINOv2) to allow generalization to new objects.
  • Figure 3: Changes in success rates (%) with design choices for GROOT.
  • Figure 4: Success rates (%) of GROOT in real-robot tasks.
  • Figure 5: Overview of objects used in real-robot experiments. In each image, the single object on the left side is used during data collection, and all the objects on the right side are not seen during training. They are used for the evaluation of new object generalization in each task.
  • ...and 2 more figures