3D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentation

TL;DR

Open-vocabulary segmentation for LiDAR point clouds is challenging due to unknown, scene-specific vocabularies. 3D-AVS introduces a CLIP-aligned pipeline that auto-generates a per-scene vocabulary from both image- and LiDAR-based captions, parses nouns into semantic tags, and assigns point labels via a CLIP-based similarity framework, complemented by Sparse Masked Attention Pooling (SMAP) for diverse object coverage. A novel Text-Point Semantic Similarity (TPSS) metric enables annotation-free evaluation of semantic alignment between points and generated labels, while an LLM-based mapper (LAVE) connects auto-generated vocabularies to fixed-ground-truth categories for standard metrics. Evaluations on nuScenes and ScanNet200 show that 3D-AVS produces semantically richer and more accurate segmentations than predefined vocabularies, including fine-grained labels in challenging lighting, reflecting strong scalability and practical impact for open-ended 3D perception.

Abstract

Open-Vocabulary Segmentation (OVS) methods offer promising capabilities in detecting unseen object categories, but the category must be known and needs to be provided by a human, either via a text prompt or pre-labeled datasets, thus limiting their scalability. We propose 3D-AVS, a method for Auto-Vocabulary Segmentation of 3D point clouds for which the vocabulary is unknown and auto-generated for each input at runtime, thus eliminating the human in the loop and typically providing a substantially larger vocabulary for richer annotations. 3D-AVS first recognizes semantic entities from image or point cloud data and then segments all points with the automatically generated vocabulary. Our method incorporates both image-based and point-based recognition, enhancing robustness under challenging lighting conditions where geometric information from LiDAR is especially valuable. Our point-based recognition features a Sparse Masked Attention Pooling (SMAP) module to enrich the diversity of recognized objects. To address the challenges of evaluating unknown vocabularies and avoid annotation biases from label synonyms, hierarchies, or semantic overlaps, we introduce the annotation-free Text-Point Semantic Similarity (TPSS) metric for assessing generated vocabulary quality. Our evaluations on nuScenes and ScanNet200 demonstrate 3D-AVS's ability to generate semantic classes with accurate point-wise segmentations. Codes will be released at https://github.com/ozzyou/3D-AVS

Figures (8)

• Figure 1: Pre-defined Vocabulary vs. Auto-Vocabulary. 3D-AVS automatically generates a vocabulary for which it predicts segmentation masks, offering greater semantic precision. Our predictions identify specific classes e.g.building and signboard (highlighted in red boxes), which are annotated with ambiguous terms like manmade. Quantitatively, 3D-AVS recognizes 671 unique categories on the validation set of nuScenes nuscenes, significantly surpassing nuScenes's original 16 categories. Left: Vocabulary for a single scene, Right: Vocabulary for the entire dataset.
• Figure 2: Overview of 3D-AVS. A point cloud and corresponding images are fed to respective point captioner and image captioner to generate captions. Then, Caption2Tag excludes irrelevant words in the captions. The remaining nouns are passed to a text encoder and eventually assigned to points through a segmenter. The dashed lines indicate that the entire images branch is optional. The point captioner is the only trainable component in 3D-AVS. Note that, the example point caption is generated based on observing the green points.
• Figure 3: Point Captioner Overview. The image encoder and point encoder are pre-aligned in the CLIP latent space. During training (left), Sparse Masked Attention Pooling (SMAP) aggregates features from points visible in the image (highlighted in red) and is supervised using CLIP image features. During inference (right), neither the image nor camera intrinsic parameters are available. To address this, a group of masks are generated based solely on geometric information. The SMAP output is then decoded into a group of captions. For simplicity, only one image (left) and one sector (right) are shown.
• Figure 4: Sparse Masked Attention Pooling (SMAP). Given the coordinates and features of all points, a relative positional encoding (PE) is applied, followed by a residual connection. Masks are applied to the points, creating groups of point subsets. Global Average Pooling (GAP) on each subset produces a mean feature as a query. Finally, multi-head attention (MHA) is applied within each group to generate one feature per subset.
• Figure 5: TPSS on nuScenes subsets with different light conditions. LiDAR-only 3D-AVS performs better during night and rainy scenes, suggesting its robustness across difficult conditions.