SegGraph: Leveraging Graphs of SAM Segments for Few-Shot 3D Part Segmentation

TL;DR

The paper tackles few-shot 3D part segmentation by leveraging SAM-generated segment masks to preserve and propagate geometric structure in 3D. It introduces SegGraph, a segment-graph propagation framework that pools multi-view 2D foundation-model features, constructs a SAM-segment graph with overlapping and adjacency relations, and refines segment representations through GATv2, followed by view-quality-aware unpooling to emit point-wise labels. This approach yields substantial gains on PartNet-E and robust improvements across various foundation models, especially for small parts and boundary regions. The results demonstrate strong geometry-aware 2D-to-3D knowledge transfer with limited annotations and highlight SegGraph's potential for practical few-shot 3D segmentation tasks.

Abstract

This work presents a novel framework for few-shot 3D part segmentation. Recent advances have demonstrated the significant potential of 2D foundation models for low-shot 3D part segmentation. However, it is still an open problem that how to effectively aggregate 2D knowledge from foundation models to 3D. Existing methods either ignore geometric structures for 3D feature learning or neglects the high-quality grouping clues from SAM, leading to under-segmentation and inconsistent part labels. We devise a novel SAM segment graph-based propagation method, named SegGraph, to explicitly learn geometric features encoded within SAM's segmentation masks. Our method encodes geometric features by modeling mutual overlap and adjacency between segments while preserving intra-segment semantic consistency. We construct a segment graph, conceptually similar to an atlas, where nodes represent segments and edges capture their spatial relationships (overlap/adjacency). Each node adaptively modulates 2D foundation model features, which are then propagated via a graph neural network to learn global geometric structures. To enforce intra-segment semantic consistency, we map segment features to 3D points with a novel view-direction-weighted fusion attenuating contributions from low-quality segments. Extensive experiments on PartNet-E demonstrate that our method outperforms all competing baselines by at least 6.9 percent mIoU. Further analysis reveals that SegGraph achieves particularly strong performance on small components and part boundaries, demonstrating its superior geometric understanding. The code is available at: https://github.com/YueyangHu2000/SegGraph.

Figures (8)

• Figure 1: Architecture comparisons of different methods. (a) 3D label aggregation-based methods liu2023partslipabdelreheem2023satrzhou2023partslip++kim2024partstadthai20243, (b) 3D feature aggregation-based methods cops2024cvprGeoZe, (c) distillation-based methods umam2024partdistillSAMPart3Dliu2025partfield to distill 2D knowledge from foundation models, and (d) ours with an emphasis on SAM segments and SAM-segments-based feature propagation.
• Figure 2: The overview of the pipeline. Given a 3D point cloud, we render it into multi-view images. We extract individual point features by pooling foundation model features (DINOv2) of rendering views. At the same time, multi-view images are fed to SAM for segment generation. Point features and segments are sent to a segment graph to learn geometric features and segment relations for part prediction.
• Figure 3: An illustration of graph construction. Features of 3D points (green dots) are first aggregated to segment nodes (blue boxes).
• Figure 4: Two examples for the impacts of the rendering views on the quality of SAM segments. For each group, left: input, right: SAM segments.
• Figure 5: Qualitative comparison of part segmentation results on the PartNet-E dataset under the few-shot setting.