Better Call SAL: Towards Learning to Segment Anything in Lidar

TL;DR

SAL introduces a promptable zero-shot Lidar segmentation framework that leverages a pseudo-label engine to distill vision foundation models into Lidar data, enabling segmentation and zero-shot classification without manual labels. A Minkowski-Transformer-based zero-shot model learns from pseudo-labels, producing class-agnostic masks and CLIP-style tokens for arbitrary text prompts, with inference requiring no image features. On SemanticKITTI and nuScenes, SAL achieves about $91\%$ of GT class-agnostic segmentation and up to $54\%$ of fully supervised LPS in zero-shot settings, while enabling segmentation of objects outside predefined vocabularies. The approach demonstrates strong potential for scalable, promptable Lidar perception, albeit with remaining gaps to fully supervised performance and cross-sensor generalization, which can be mitigated by data scaling and temporal-context integration.

Abstract

We propose the SAL (Segment Anything in Lidar) method consisting of a text-promptable zero-shot model for segmenting and classifying any object in Lidar, and a pseudo-labeling engine that facilitates model training without manual supervision. While the established paradigm for Lidar Panoptic Segmentation (LPS) relies on manual supervision for a handful of object classes defined a priori, we utilize 2D vision foundation models to generate 3D supervision ``for free''. Our pseudo-labels consist of instance masks and corresponding CLIP tokens, which we lift to Lidar using calibrated multi-modal data. By training our model on these labels, we distill the 2D foundation models into our Lidar SAL model. Even without manual labels, our model achieves $91\%$ in terms of class-agnostic segmentation and $54\%$ in terms of zero-shot Lidar Panoptic Segmentation of the fully supervised state-of-the-art. Furthermore, we outperform several baselines that do not distill but only lift image features to 3D. More importantly, we demonstrate that SAL supports arbitrary class prompts, can be easily extended to new datasets, and shows significant potential to improve with increasing amounts of self-labeled data. Code and models are available at this $\href{https://github.com/nv-dvl/segment-anything-lidar}{URL}$.

Figures (15)

• Figure 1: The SAL (Segment Anything in Lidar) model performs class-agnostic instance segmentation (i) and zero-shot classification via text prompting. This allows us to not only predict panoptic segmentation (ii) for fixed class vocabularies but also segment any object (iii and iv) in a given Lidar scan.
• Figure 2: SAL overview: Given a Lidar scan and a class vocabulary prompt, specified as a list of per-class free-form text descriptions (left), SAL segments and classifies objects (things and stuff classes). As labeled data for training such a model does not exist, we supervise SAL by distilling off-the-shelf vision foundation models to Lidar (right).
• Figure 3: Our pseudo-label engine (\ref{['fig:pseudo-engine']}) utilizes SAM kirillov2023segment to estimate segmentation masks in images, MaskCLIP ding2023open to estimate corresponding per-mask CLIP features, and a calibrated sensory setup to transfer them to the Lidar domain. We distill these pseudo-labels to our zero-shot model (\ref{['fig:zero-shot-model']}), which segments and classifies Lidar point clouds. The SAL zero-shot model employs a sparse-convolutional backbone choy20194d, followed by a Transformer decoder that predicts objectness scores, segmentation masks, and CLIP tokens for each query. To (optionally) perform zero-shot classification, we forward the dataset class vocabulary through the CLIP text encoder and match the encoded vocabulary with predicted CLIP tokens. Our model requires no retraining for different vocabularies and no image features at inference time.
• Figure 4: Refinement via clustering. After transferring image masks (\ref{['fig:sam']}) to Lidar (\ref{['fig:sam_unproj']}), we obtain pseudo-labels that suffer from sensory misalignment-related issues. Our geometric refinement (\ref{['fig:sam_unproj_refined']}) improves localization.
• Figure A.1: SAL zero-shot model. Our model takes Lidar point clouds and text prompts as inputs. Its architecture relies on a 3D sparse-convolutional Minkowski backbone choy20194d followed by a Transformer decoder for object instance segmentation. The decoder computes cross-attention between object queries and backbone features. Three task heads predict objectness scores, segmentation masks, and CLIP tokens for each query. Once trained, we forward the dataset class vocabulary through the CLIP text encoder and perform zero-shot classification via matching with predicted CLIP tokens. The model requires no retraining for different vocabularies.