UniLiPs: Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition

TL;DR

UniLiPs tackles the annotation bottleneck in autonomous driving by introducing an unsupervised unified 3D labeling framework that leverages temporal-geometric consistency across LiDAR sweeps to fuse text-and-vision foundation-model cues into 3D. The geometry-grounded approach decomposes scenes into static and dynamic components, propagates semantic labels from images into a consolidated 3D map, and iteratively refines both geometry and semantics using an Iterative Weighted Update. The method jointly produces 3D semantic labels, 3D bounding boxes, and densified LiDAR scans, generalizing across KITTI, nuScenes, and long-range highway data, and achieves near-oracle semantic labeling while markedly improving depth estimation up to $250$ meters, e.g., MAE reductions of $51.5\%$ in $80$-$150$ m and $22.0\%$ in $150$-$250$ m ranges. This work reduces annotation effort and enables robust long-range perception by delivering reliable pseudo-labels for multiple downstream tasks without manual supervision.

Abstract

Unlabeled LiDAR logs, in autonomous driving applications, are inherently a gold mine of dense 3D geometry hiding in plain sight - yet they are almost useless without human labels, highlighting a dominant cost barrier for autonomous-perception research. In this work we tackle this bottleneck by leveraging temporal-geometric consistency across LiDAR sweeps to lift and fuse cues from text and 2D vision foundation models directly into 3D, without any manual input. We introduce an unsupervised multi-modal pseudo-labeling method relying on strong geometric priors learned from temporally accumulated LiDAR maps, alongside with a novel iterative update rule that enforces joint geometric-semantic consistency, and vice-versa detecting moving objects from inconsistencies. Our method simultaneously produces 3D semantic labels, 3D bounding boxes, and dense LiDAR scans, demonstrating robust generalization across three datasets. We experimentally validate that our method compares favorably to existing semantic segmentation and object detection pseudo-labeling methods, which often require additional manual supervision. We confirm that even a small fraction of our geometrically consistent, densified LiDAR improves depth prediction by 51.5% and 22.0% MAE in the 80-150 and 150-250 meters range, respectively.

Figures (5)

• Figure 1: Unified 3D Labeling. Given a single driving trajectory, UniLiPs fuse consecutive LiDAR scans with our engine’s 2D pseudo-labels to build a coherent 3D map. Within this consistent geometry, moving actors and semantic labels are optimized to jointly generate refined, temporally consistent 3D bounding boxes, semantic labels, and occlusion-aware, densified point clouds.
• Figure 2: Overview of Geometry-Grounded Dynamic Scene Decomposition. Starting from a set of raw images, a set of LiDAR scan and IMU measurements, we first produce 3D semantic labels. Therefore, the 2D semantic masks produced by $f_{\text{seg}}$ are integrated into a map generated by the SLAM method $f_{\text{map}}$, by projecting them through $P_{\text{cam}}$, while simultaneously removing moving points identified by $f_{\text{mos}}$ from the map. To obtain a refined static scene map $\mathcal{M}_{refined}$ we first propagate the labels through geometric and temporal constraints and later on exploit them to remove remaining floaters and outliers (in red) through a our Iterative Weighted Update Function$f_{IWU}$.
• Figure 3: Overview of Pseudo Labeling Function. Our proposed pseudo labeling method $f_{seg}$ robustly segments each $2D$ image $I_t$ by combining the predictions from an ensamble of three OneFormer jain2022oneformertransformerruleuniversal models with weights from three different datasets (COCO lin2015microsoftcococommonobjects, ADE20K zhou2017scenezhou2019semantic and Cityscapes Cordts_2016_CVPR) and a SAM2 ravi2024sam2 instance prediction set $Mask_t$. For each mask $m_i$, BLIP li2022blip enriches the class proposals and through modal alignment with CLIP radford2021learningtransferablevisualmodels and CLIPSeg lueddecke22_cvpr, we ensure high quality domain-specific annotations.
• Figure 4: UniLiPs Unified Labeling Outputs. Coupling geometric point-cloud aggregation with image segmentation cues from our $f_{seg}$, UniLiPs rivals standalone methods by jointly producing temporally consistent semantic labels, trajectory-smoothed bounding boxes, and densified LiDAR sweeps that are denser and offer finer angular resolution, especially at long range. In the Figure, densified LiDARs are z-colored between $-2$ (blue) and $+5$ (red) meters, while semantics are class-coloured based on SemanticKITTI mapping.
• Figure 5: Effect of Semantic Multimodal Propagation. Leveraging our refined, geometry-grounded map as a reference, mislabeled points in each LiDAR scan are systematically corrected, ensuring label consistency across all timestamps.