Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery

Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song

TL;DR

SfA presents a unified framework that learns to interact with unknown articulated objects to uncover their 3D part geometry and joints. By coupling a 3D action policy with a persistent perception module and a joint inference component, SfA produces complete articulated CAD models and generalizes to unseen categories and real objects. The approach demonstrates clear advantages over state-of-the-art pipelines in both simulation and real-world settings, highlighting the importance of integrating interaction and perception. This work enables autonomous discovery of object structure for manipulation and planning in unstructured environments.

Abstract

We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially for categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a SfA model trained in simulation can generalize to many unseen object categories with diverse structures and to real-world objects. Empirically, SfA outperforms a pipeline of state-of-the-art components by 25.4 3D IoU percentage points on unseen categories, while matching already performant joint estimation baselines.

Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery

TL;DR

SfA presents a unified framework that learns to interact with unknown articulated objects to uncover their 3D part geometry and joints. By coupling a 3D action policy with a persistent perception module and a joint inference component, SfA produces complete articulated CAD models and generalizes to unseen categories and real objects. The approach demonstrates clear advantages over state-of-the-art pipelines in both simulation and real-world settings, highlighting the importance of integrating interaction and perception. This work enables autonomous discovery of object structure for manipulation and planning in unstructured environments.

Abstract

We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially for categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a SfA model trained in simulation can generalize to many unseen object categories with diverse structures and to real-world objects. Empirically, SfA outperforms a pipeline of state-of-the-art components by 25.4 3D IoU percentage points on unseen categories, while matching already performant joint estimation baselines.
Paper Structure (10 sections, 8 figures, 3 tables)

This paper contains 10 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structure From Action
  4. Learning to Interact with Articulated Parts
  5. Learning Persistent Part Aggregation
  6. Recovering Joints
  7. Constructing an Articulated CAD Model
  8. Experiments
  9. Experimental Results
  10. Conclusion

Figures (8)

  • Figure 1: Structure from Action. Our framework discovers an object's structure through a sequence of 3D interactions. The resulting structure includes a) part segmentation, b) 3D reconstruction for each part, and c) joint parameters, together describing d) a 3D articulated CAD model.
  • Figure 2: Overview. Given an RGB point cloud observation of an unknown articulated object, SfA infers and executes a sequence of informative actions (§ \ref{['sec:policy']}), discovers and reconstructs parts (§ \ref{['sec:part']}), estimates joint parameters (§ \ref{['sec:joints']}), and outputs an articulated 3D CAD model of the object (§ \ref{['sec:urdf']}).
  • Figure 3: Learning Interaction Policy.(left) During training, the 3D scene flow is used to supervise the action directions (green). For a timestep, areas where flow is zero are assumed to be good hold locations (red). (right) Inferred candidate push actions conditioned on a sampled hold action.
  • Figure 4: Dynamic part reconstruction. SfA completes part geometry by aggregating all past observations in a spatially consistent manner.
  • Figure 5: Joint Inference.(left) Revolute joint position and axis orientation votes. (right) Prismatic joint orientation votes.
  • ...and 3 more figures