Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model

TL;DR

This paper tackles generalized few-shot 3D point cloud segmentation by leveraging the open-world knowledge embedded in 3D Vision-Language Models to provide dense pseudo-labels for novel classes, while relying on precise, sparse $K$-shot support data to ground learning. It introduces three key innovations: pseudo-label selection to filter noisy VLM predictions, adaptive infilling to label unlabeled regions using adaptive prototypes, and novel-base mix to embed few-shot samples into training scenes while preserving contextual cues. To address evaluation gaps, it also proposes two challenging benchmarks (ScanNet200 and ScanNet++) with broader novel-class diversity, enabling more realistic generalization assessment. Across multiple backbones and datasets, the approach achieves state-of-the-art performance, demonstrating the practical potential of combining 3D VLM knowledge with few-shot cues for robust open-world 3D scene understanding, and its accompanying benchmarks provide a solid foundation for future work.

Abstract

Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to new classes with few support samples while retaining base class segmentation. Existing GFS-PCS methods enhance prototypes via interacting with support or query features but remain limited by sparse knowledge from few-shot samples. Meanwhile, 3D vision-language models (3D VLMs), generalizing across open-world novel classes, contain rich but noisy novel class knowledge. In this work, we introduce a GFS-PCS framework that synergizes dense but noisy pseudo-labels from 3D VLMs with precise yet sparse few-shot samples to maximize the strengths of both, named GFS-VL. Specifically, we present a prototype-guided pseudo-label selection to filter low-quality regions, followed by an adaptive infilling strategy that combines knowledge from pseudo-label contexts and few-shot samples to adaptively label the filtered, unlabeled areas. Additionally, we design a novel-base mix strategy to embed few-shot samples into training scenes, preserving essential context for improved novel class learning. Moreover, recognizing the limited diversity in current GFS-PCS benchmarks, we introduce two challenging benchmarks with diverse novel classes for comprehensive generalization evaluation. Experiments validate the effectiveness of our framework across models and datasets. Our approach and benchmarks provide a solid foundation for advancing GFS-PCS in the real world. The code is at https://github.com/ZhaochongAn/GFS-VL

Figures (8)

• Figure 1: Comparison of our framework with previous work.Top: Prior work tian2022generalizedxu2023generalized primarily enhances prototypes through interaction modules that integrate support/query features, making predictions based on refined prototypes. However, they are limited by the sparse knowledge from few-shot samples. Bottom: Our framework addresses this limitation by leveraging the extensive open-world knowledge from 3D VLMs through pseudo-labels. We mitigate the noise inherent in 3D VLMs by calibrating their raw pseudo-labels with precise few-shot samples, thereby effectively expanding novel class knowledge while ensuring reliability.
• Figure 2: Overview of the proposed GFS-VL. (a), (b) Given an input point cloud $\mathbf{X}_{\rm b}$, we apply a novel-base mix to embed support samples into the training scene while preserving essential context. The scene is then processed by a 3D VLM, using all class names as prompts to generate raw predictions $\mathbf{\hat{Y}}$. Leveraging support prototypes $\{\mathbf{p}^{\rm c}\}$, the raw predictions undergo pseudo-label selection to filter out noisy regions, followed by adaptive infilling to label the filtered, unlabeled areas, yielding refined supervision $\mathbf{Y}^{\rm \prime\prime}_{\rm b}$ for training the 3D segmentor. (c), (d) illustrate the details of the pseudo-label selection and adaptive infilling processes.
• Figure 3: Qualitative comparison between GW xu2023generalized and our GFS-VL on ScanNet200. Class colors are shown at the top.
• Figure 4: Visualization of the improvements in pseudo-label quality after applying Pseudo-label Selection (PS) and Adaptive Infilling (AI). Note that AI effectively discovers missed novel classes in the red circles and completes partial pseudo-labels in the green circles.
• Figure 5: Visual illustration of mixing strategies. The red and green boxes represent the two novel samples mixed into the scene.