PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

TL;DR

PartGen addresses the need for structured, editable 3D assets by decomposing objects into meaningful parts using dual diffusion priors: one for part segmentation and one for context-aware part completion, followed by a 3D reconstruction step. The method handles ambiguity by sampling multiple plausible segmentations and by completing occluded parts with global context, enabling coherent assembly and part editing. It demonstrates improvements over baselines in segmentation, completion, and reconstruction on artist-created and real-world assets, and shows practical applications in part-aware text- and image-to-3D generation as well as real-world decomposition and editing. This work advances compositional 3D generation by providing controllable, reusable parts suitable for editing, animation, and downstream tasks in professional workflows.

Abstract

Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.

Figures (12)

• Figure 1: We introduce PartGen, a pipeline that generates compositional 3D objects similar to a human artist. It can start from text, an image, or an existing, unstructured 3D object. It consists of a multi-view diffusion model that identifies plausible parts automatically and another that completes and reconstructs them in 3D, accounting for their context, i.e., the other parts, to ensure that they fit together correctly. Additionally, PartGen enables 3D part editing based on text instructions, enhancing flexibility and control in 3D object creation.
• Figure 2: Overview of PartGen. Our method begins with text, single images, or existing 3D objects to obtain an initial grid view of the object. This view is then processed by a diffusion-based segmentation network to achieve multi-view consistent part segmentation. Next, the segmented parts, along with contextual information, are input into a multi-view part completion network to generate a fully completed view of each part. Finally, a pre-trained reconstruction model generates the 3D parts.
• Figure 3: Training data. We obtain a dataset of 3D objects decomposed into parts from assets created by artists. These come 'naturally' decomposed into parts according to the artist's design.
• Figure 4: Examples of automatic multi-view part segmentations. By running our method several times, we obtain different segmentations, covering the space of artist intents.
• Figure 5: Qualitative results of part completion. The images with blue borders are the inputs. Our algorithm produces various plausible outputs across different runs. Even if given an empty part, PartGen attempts to generate internal structures inside the object, such as sand or inner wheels.