Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Oriented Hierarchical Object Decomposition for Visuomotor Control

Jianing Qian, Yunshuang Li, Bernadette Bucher, Dinesh Jayaraman

TL;DR

HODOR is proposed to train a large combinatorial family of representations organized by scene entities: objects and object parts that permits selectively assembling different representations specific to each task while scaling in representational capacity with the complexity of the scene and the task.

Abstract

Good pre-trained visual representations could enable robots to learn visuomotor policy efficiently. Still, existing representations take a one-size-fits-all-tasks approach that comes with two important drawbacks: (1) Being completely task-agnostic, these representations cannot effectively ignore any task-irrelevant information in the scene, and (2) They often lack the representational capacity to handle unconstrained/complex real-world scenes. Instead, we propose to train a large combinatorial family of representations organized by scene entities: objects and object parts. This hierarchical object decomposition for task-oriented representations (HODOR) permits selectively assembling different representations specific to each task while scaling in representational capacity with the complexity of the scene and the task. In our experiments, we find that HODOR outperforms prior pre-trained representations, both scene vector representations and object-centric representations, for sample-efficient imitation learning across 5 simulated and 5 real-world manipulation tasks. We further find that the invariances captured in HODOR are inherited into downstream policies, which can robustly generalize to out-of-distribution test conditions, permitting zero-shot skill chaining. Appendix, code, and videos: https://sites.google.com/view/hodor-corl24.

Task-Oriented Hierarchical Object Decomposition for Visuomotor Control

TL;DR

HODOR is proposed to train a large combinatorial family of representations organized by scene entities: objects and object parts that permits selectively assembling different representations specific to each task while scaling in representational capacity with the complexity of the scene and the task.

Abstract

Good pre-trained visual representations could enable robots to learn visuomotor policy efficiently. Still, existing representations take a one-size-fits-all-tasks approach that comes with two important drawbacks: (1) Being completely task-agnostic, these representations cannot effectively ignore any task-irrelevant information in the scene, and (2) They often lack the representational capacity to handle unconstrained/complex real-world scenes. Instead, we propose to train a large combinatorial family of representations organized by scene entities: objects and object parts. This hierarchical object decomposition for task-oriented representations (HODOR) permits selectively assembling different representations specific to each task while scaling in representational capacity with the complexity of the scene and the task. In our experiments, we find that HODOR outperforms prior pre-trained representations, both scene vector representations and object-centric representations, for sample-efficient imitation learning across 5 simulated and 5 real-world manipulation tasks. We further find that the invariances captured in HODOR are inherited into downstream policies, which can robustly generalize to out-of-distribution test conditions, permitting zero-shot skill chaining. Appendix, code, and videos: https://sites.google.com/view/hodor-corl24.

Paper Structure

This paper contains 25 sections, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pre-trained representations in computer vision and robotics
  4. Object-centric embeddings (OCE)
  5. Hierarchical coarse-to-fine representations
  6. Task conditioning in policies and representations
  7. Inferring HODOR Representations From Foundation Models
  8. Preliminaries: Object-Centric Image Embeddings from Pre-trained Foundation Models
  9. Task-Conditioned Entity Selection
  10. Hierarchical Object Decomposition
  11. Policy Architecture and Training
  12. Experiments
  13. Simulated Experiments
  14. Real Robot Experiments
  15. Conclusions and Limitations
  16. ...and 10 more sections

Figures (13)

  • Figure 1: HODOR embeds a 2D input image into an object-centric, multi-resolution, task-specific representation for efficient policy learning
  • Figure 2: Policy Architecture. We illustrated the resulting HODOR representations for task Put Eggplant in Pot. There are two relevant objects in this task: eggplant and pot. Their corresponding parts are also visualized.
  • Figure 3: Task Visualization.
  • Figure 4: Visualization of simulated Franka Kitchen tasks.
  • Figure 5: BC performance of HODOR and baselines as a function of the number of demonstrations measured by success rate on five tasks from the FrankaKitchen benchmark. Shaded regions show the standard error across 3 seeds.
  • ...and 8 more figures