Towards Learning to Complete Anything in Lidar

TL;DR

This paper tackles zero-shot LiDAR scene completion by learning shape priors and open-vocabulary semantics from unlabeled, temporally rich multi-modal data. It introduces CAL, a two-part approach: a pseudo-labeling engine that mines 3D shape priors and CLIP-based semantics from video and a zero-shot, class-agnostic completion model built on a sparse 3D U-Net and a Transformer decoder, capable of SSC, PSC, or amodal detection with test-time vocabulary prompts. The key contributions include a scalable pseudo-labeling pipeline using vision foundation models, CLIP-derived semantic prototypes for open vocabularies, and a voxel-based completion framework that generalizes beyond fixed class vocabularies. Results on SemanticKITTI and SSCBench-KITTI360 show competitive zero-shot PSC/SSC performance and reveal both the promise and current limitations of open-vocabulary LiDAR perception, with significant room for improvement in label coverage and recognition under zero-shot conditions.

Abstract

We propose CAL (Complete Anything in Lidar) for Lidar-based shape-completion in-the-wild. This is closely related to Lidar-based semantic/panoptic scene completion. However, contemporary methods can only complete and recognize objects from a closed vocabulary labeled in existing Lidar datasets. Different to that, our zero-shot approach leverages the temporal context from multi-modal sensor sequences to mine object shapes and semantic features of observed objects. These are then distilled into a Lidar-only instance-level completion and recognition model. Although we only mine partial shape completions, we find that our distilled model learns to infer full object shapes from multiple such partial observations across the dataset. We show that our model can be prompted on standard benchmarks for Semantic and Panoptic Scene Completion, localize objects as (amodal) 3D bounding boxes, and recognize objects beyond fixed class vocabularies. Our project page is https://research.nvidia.com/labs/dvl/projects/complete-anything-lidar

Figures (8)

• Figure 1: Learning to Complete Anything in Lidar. Given a sparse Lidar point cloud, CAL (Complete Anything in Lidar) localizes, reconstructs, and, optionally, recognizes objects in a zero-shot fashion. By providing a semantic class vocabulary of specific object classes at test time, CAL can be prompted to perform Semantic Scene Completion (SSC), Panoptic Scene Completion (PSC), or (amodal) 3D Object Detection. Note that CAL only takes a single Lidar scan as input; RGB images are shown for visualization purposes only.
• Figure 2: Pseudo-labeling engine. Given a calibrated RGB camera and Lidar sensor, ① we use video-object segmentation models ravi2024sam2 to localize object instances in video, ② pseudo-label the Lidar point clouds over time, and ③ generate completed voxelized object representations, each enriched with a per-instance CLIP feature extracted from RGB images. In ④, we accumulate 360$^\circ$ Lidar scans to obtain full-scene binary occupancy, used for refining the aggregated pseudo-labels ③ via a CRF-guided label refinement process ⑤. As output ⑥, our method pairs each sparse and incomplete Lidar scan with pseudo-labels for object-level scene completion (top-right) and CLIP features, which are temporally aggregated by averaging per-instance features across the sequence. These CLIP features enable zero-shot recognition via text queries (bottom-right). Mined pseudo-label pairs are then used to train the CAL model.
• Figure 3: CAL model architecture and training pipeline. The backbone consists of a sparse encoder and a dense 3D convolutional block. We estimate scene-level occupancy using a multi-scale sparse generative decoder that consists of decoder blocks $D$, two occupancy heads $B_o$ and $B_s$, and a pseudo-semantic head ($S$) at each scale $\mathrm{L}$. The Transformer decoder then predicts segmentation masks over the completed scene and regresses CLIP features.
• Figure 4: Qualitative results on SemanticKITTI. Given a single Lidar scan (1$^{st}$ col.), CAL completes object-level observations as a set of masks over the voxel grid (2$^{nd}$ col.) and predicts a CLIP feature for each mask. We can prompt with any semantic class vocabulary and obtain panoptic and semantic scene completion (3$^{rd}$ col.) results. Our model predicts shape priors for both thing (e.g., 'car', 'cyclist') and stuff classes (e.g. 'vegetation', 'road') and can correctly predict the intersection geometry in 4$^{th}$ row, despite limited direct evidence.
• Figure 5: Completion and amodal detection on KITTI-360. Given an input Lidar scan (left), CAL outputs a set of completed object shapes (middle). We visualize recognized objects (right) for queries 'vehicle' (top), 'car' (middle) and 'tree' (bottom), and fit 3D bounding boxes to the identified object instances, demonstrating the zero-shot amodal 3D object detection ability of CAL.