SegviGen: Repurposing 3D Generative Model for Part Segmentation

Abstract

We introduce SegviGen, a framework that repurposes native 3D generative models for 3D part segmentation. Existing pipelines either lift strong 2D priors into 3D via distillation or multi-view mask aggregation, often suffering from cross-view inconsistency and blurred boundaries, or explore native 3D discriminative segmentation, which typically requires large-scale annotated 3D data and substantial training resources. In contrast, SegviGen leverages the structured priors encoded in pretrained 3D generative model to induce segmentation through distinctive part colorization, establishing a novel and efficient framework for part segmentation. Specifically, SegviGen encodes a 3D asset and predicts part-indicative colors on active voxels of a geometry-aligned reconstruction. It supports interactive part segmentation, full segmentation, and full segmentation with 2D guidance in a unified framework. Extensive experiments show that SegviGen improves over the prior state of the art by 40% on interactive part segmentation and by 15% on full segmentation, while using only 0.32% of the labeled training data. It demonstrates that pretrained 3D generative priors transfer effectively to 3D part segmentation, enabling strong performance with limited supervision. See our project page at https://fenghora.github.io/SegviGen-Page/.

Figures (8)

• Figure 1: SegviGen enables diverse and accurate 3D part segmentation by leveraging priors from large-scale 3D generative models. With substantially less training data, it produces high-fidelity segmentation results with sharp part boundaries and strong generalization across object categories.
• Figure 2: Pipeline of SegviGen. We reformulate 3D part segmentation as a conditional colorization task. During training, given a 3D mesh and its part-color ground truth, we encode both with a pretrained 3D VAE, add noise to the ground-truth latent, and then concatenate the geometry latent, noisy color latent, and point-condition tokens to form the final latent input. Conditioned on the sampled timestep and a task embedding, the multi-task flow transformer predicts the noise residual for flow-matching training.
• Figure 3: Interactive part-segmentation results. We compare SegviGen with existing representative baselines, including Point-SAM pointsam and P3-SAM p3sam. In the figure, yellow points denote user clicks, while the predicted target part is highlighted in red. Leveraging priors from pretrained 3D generative models, SegviGen achieves more accurate results with sharper boundaries than prior methods, while requiring substantially less training data.
• Figure 4: Full segmentation results. We compare SegviGen against a broad set of prior methods, where different colors indicate different segmented parts. From the results, SegviGen achieves high-accuracy full segmentation with sharp part boundaries using only 3D input. When 2D guidance is provided, it further allows explicit control over granularity and labels, enabling controllable, ultra-fine-grained 3D part parsing.
• Figure 5: More qualitative segmentation results of our SegviGen.