PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond

TL;DR

PartField tackles open-world 3D part segmentation by learning a fast, 3D-native feature field through a feedforward model and a contrastive triplet objective trained on diverse 2D and 3D part proposals. The resulting triplane feature representation can be clustered into hierarchical parts and used for co-segmentation and correspondences, achieving significant accuracy gains (over 20% mIoU) with real-time inference. The approach demonstrates cross-modality robustness, scalable training with large datasets, and useful downstream capabilities, including interactive segmentation and cross-shape correspondence, highlighting its practical impact for open-world 3D shape understanding.

Abstract

We propose PartField, a feedforward approach for learning part-based 3D features, which captures the general concept of parts and their hierarchy without relying on predefined templates or text-based names, and can be applied to open-world 3D shapes across various modalities. PartField requires only a 3D feedforward pass at inference time, significantly improving runtime and robustness compared to prior approaches. Our model is trained by distilling 2D and 3D part proposals from a mix of labeled datasets and image segmentations on large unsupervised datasets, via a contrastive learning formulation. It produces a continuous feature field which can be clustered to yield a hierarchical part decomposition. Comparisons show that PartField is up to 20% more accurate and often orders of magnitude faster than other recent class-agnostic part-segmentation methods. Beyond single-shape part decomposition, consistency in the learned field emerges across shapes, enabling tasks such as co-segmentation and correspondence, which we demonstrate in several applications of these general-purpose, hierarchical, and consistent 3D feature fields. Check our Webpage! https://research.nvidia.com/labs/toronto-ai/partfield-release/

Figures (13)

• Figure 1: We propose PartField, a feedforward model that predicts part-based feature fields for 3D shapes. The learned features can be clustered to yield a high-quality part decomposition, and our method outperforms the latest open-world 3D part segmentation approaches in both quality and speed. PartField can be applied to a wide variety of inputs in terms of modality, semantic class, and style. The learned feature field exhibits consistency across shapes, enabling applications such as cosegmentation, interactive selection, and correspondence.
• Figure 2: PartField part segmentations across various 3D input modalities.
• Figure 3: We train a feedforward model that takes a point-sampled 3D shape as input (which could come from a mesh, Gaussian splats, or other representations) and predicts a feature field represented by a triplane. These features can then be clustered to generate parts at various scales. Our model is trained with a contrastive loss on both open-world data, distilled from image-space masks, which need not be consistent, and 3D supervision when available.
• Figure 4: (Left) A point can belong to multiple parts at different scales. (Upper Right) Prior works garfield2024yang2024sampart3d utilize pull and push losses to directly minimize or maximize the feature distances between point pairs, requiring an additional scaling condition to learn point features at different scales. (Lower Right) Our method employs a triplet loss that only encourages the relative relations between points within a triplet, enabling self-contained features ($\text{sim}(f(A),f(B)) > \text{sim}(f(A),f(C)) > \text{sim}(f(A),f(D))$) that support multi-scale parts without need of scaling condition.
• Figure 5: Qualitative comparison of class-agnostic segmentation on the PartObjaverse-Tiny dataset yang2024sampart3d. The baselines include Find3D find3d, PartSLIP liu2023partslip, Ultrametric Feature Field ultrametric, SAMesh tang2024segmentmeshzeroshotmesh, and SAMpart3D yang2024sampart3d. Each color represents a different part.