Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing

TL;DR

A part-whole-hierarchy message passing network for efficient 3D part assembly that achieves state-of-the-art performance in part and connectivity accuracy and enables an interpretable hierarchical part assembly.

Abstract

Generative 3D part assembly involves understanding part relationships and predicting their 6-DoF poses for assembling a realistic 3D shape. Prior work often focus on the geometry of individual parts, neglecting part-whole hierarchies of objects. Leveraging two key observations: 1) super-part poses provide strong hints about part poses, and 2) predicting super-part poses is easier due to fewer superparts, we propose a part-whole-hierarchy message passing network for efficient 3D part assembly. We first introduce super-parts by grouping geometrically similar parts without any semantic labels. Then we employ a part-whole hierarchical encoder, wherein a super-part encoder predicts latent super-part poses based on input parts. Subsequently, we transform the point cloud using the latent poses, feeding it to the part encoder for aggregating super-part information and reasoning about part relationships to predict all part poses. In training, only ground-truth part poses are required. During inference, the predicted latent poses of super-parts enhance interpretability. Experimental results on the PartNet dataset show that our method achieves state-of-the-art performance in part and connectivity accuracy and enables an interpretable hierarchical part assembly. Code is available at https://github.com/pkudba/3DHPA.

Figures (6)

• Figure 1: Left: an illustration of the part-whole-hierarchy for 3D shapes; Right: the part assembly process via the proposed part-whole-hierarchy message passing network.
• Figure 2: The overall architecture of our model consists of two modules: super-part encoder and part encoder. We first obtain super-parts via grouping parts based on their geometric similarities in an unsupervised fashion. The super-part encoder takes point cloud as input and predicts latent super-part poses (no ground truth is needed). The point cloud is then transformed based on super-part poses and fed to the part encoder. We incorporate both cross-level and within-level attention in the part encoder to predict part poses.
• Figure 3: The qualitative comparisons between our method and two most competitive baselines on PartNet mo2019partnet. We highlight some areas where the assembly quality of ours is clearly better.
• Figure 4: Diverse results on the unseen PartNetmo2019partnet test dataset generated by our network to demonstrate the structural variation in part assembly, providing different artistic results while maintaining reasonable object structures.
• Figure 5: Performance on the Chair, Table and Lamp categories under multiple Chamfer distance thresholds.