HoloPart: Generative 3D Part Amodal Segmentation

TL;DR

3D part amodal segmentation extends beyond visible surface patches by reconstructing complete semantic parts of a 3D shape under occlusion. The authors propose HoloPart, a diffusion-based part completion model with separate local and shape-context attention to capture fine details and ensure global consistency, respectively, trained atop a strong 3D generative prior and paired with existing segmentation. They validate on newly introduced ABO and PartObjaverse-Tiny benchmarks, showing significant improvements over state-of-the-art shape completion methods and strong end-to-end performance when combined with segmentation. The work enables practical 3D editing, animation, and material assignment, and sets a foundation for future 3D part-aware generation and editing tasks.

Abstract

3D part amodal segmentation--decomposing a 3D shape into complete, semantically meaningful parts, even when occluded--is a challenging but crucial task for 3D content creation and understanding. Existing 3D part segmentation methods only identify visible surface patches, limiting their utility. Inspired by 2D amodal segmentation, we introduce this novel task to the 3D domain and propose a practical, two-stage approach, addressing the key challenges of inferring occluded 3D geometry, maintaining global shape consistency, and handling diverse shapes with limited training data. First, we leverage existing 3D part segmentation to obtain initial, incomplete part segments. Second, we introduce HoloPart, a novel diffusion-based model, to complete these segments into full 3D parts. HoloPart utilizes a specialized architecture with local attention to capture fine-grained part geometry and global shape context attention to ensure overall shape consistency. We introduce new benchmarks based on the ABO and PartObjaverse-Tiny datasets and demonstrate that HoloPart significantly outperforms state-of-the-art shape completion methods. By incorporating HoloPart with existing segmentation techniques, we achieve promising results on 3D part amodal segmentation, opening new avenues for applications in geometry editing, animation, and material assignment.

Figures (15)

• Figure 1: Demonstration of the difference between (a) 3D part segmentation and (b) 3D part amodal segmentation. 3D part amodal segmentation decomposes the 3D shape into complete semantic parts rather than broken surface patches, facilitating various downstream applications. In this paper, we propose a solution by performing 3D part shape completion on incomplete part segments.
• Figure 2: An overview of the HoloPart model design. Given a whole 3D shape and a corresponding surface segmentation mask, HoloPart encodes these inputs into latent tokens, using context-aware attention to capture global shape context and local attention to capture local part detailed features and position mapping. These tokens are used as conditions and injected into the part diffusion model via cross-attention respectively. During training, noise is added to complete 3D parts, and the model learns to denoise them and recover the original complete part.
• Figure 3: Qualitative comparison with PatchComplete, DiffComplete and Finetune-VAE on the ABO dataset.
• Figure 4: Qualitative comparison with PatchComplete, DiffComplete and Finetune-VAE on the PartObjaverse-Tiny dataset.
• Figure 5: Our method seamlessly integrates with existing zero-shot 3D part segmentation models, enabling effective zero-shot 3D part amodal segmentation.