4D Contrastive Superflows are Dense 3D Representation Learners

TL;DR

This paper tackles the annotation bottleneck in autonomous-driving 3D perception by introducing SuperFlow, a dense 4D representation learner that pretrains on consecutive LiDAR–camera pairs. It combines three core ideas: view consistency alignment with language-guided semantic superpixels, dense-to-sparse regularization across multi-sweep point clouds, and flow-based contrastive learning to harness temporal cues across frames. Across 11 diverse LiDAR datasets, SuperFlow achieves state-of-the-art performance in linear probing, downstream fine-tuning, and robustness, with notable gains as 2D and 3D backbone capacities scale, suggesting emergent properties for scalable 3D foundation models. The work reduces annotation requirements while delivering robust, transferable 3D representations for LiDAR-based perception in autonomous systems.

Abstract

In the realm of autonomous driving, accurate 3D perception is the foundation. However, developing such models relies on extensive human annotations -- a process that is both costly and labor-intensive. To address this challenge from a data representation learning perspective, we introduce SuperFlow, a novel framework designed to harness consecutive LiDAR-camera pairs for establishing spatiotemporal pretraining objectives. SuperFlow stands out by integrating two key designs: 1) a dense-to-sparse consistency regularization, which promotes insensitivity to point cloud density variations during feature learning, and 2) a flow-based contrastive learning module, carefully crafted to extract meaningful temporal cues from readily available sensor calibrations. To further boost learning efficiency, we incorporate a plug-and-play view consistency module that enhances the alignment of the knowledge distilled from camera views. Extensive comparative and ablation studies across 11 heterogeneous LiDAR datasets validate our effectiveness and superiority. Additionally, we observe several interesting emerging properties by scaling up the 2D and 3D backbones during pretraining, shedding light on the future research of 3D foundation models for LiDAR-based perception.

Figures (12)

• Figure 1: Performance overview of SuperFlow compared to state-of-the-art image-to-LiDAR pretraining methods, i.e., Sealseal, SLidRslidr, and PPKTppkt, on eleven LiDAR datasets. The scores of prior methods are normalized based on SuperFlow's scores. The larger the area coverage, the better the overall segmentation performance.
• Figure 2: Comparisons of different superpixels. (a) Class-agnostic superpixels generated by the unsupervised SLIC achanta2012slic algorithm. (b) Class-agnostic semantic superpixels generated by vision foundation models (VFMs) zou2023seemzhang2023openSeeDzou2023xdecoder. (c) View-consistent semantic superpixels generated by our view consistency alignment module.
• Figure 3: Dense-to-sparse (D2S) consistency regularization module. Dense point clouds are obtained by combining multiple point clouds captured at different times. A D2S regularization is formulated by encouraging the consistency between dense features and sparse features.
• Figure 4: Flow-based contrastive learning (FCL) pipeline. FCL takes multiple LiDAR-camera pairs from consecutive scans as input. Based on temporally aligned semantic superpixel and superpoints, two contrastive learning objectives are formulated: 1) spatial contrastive learning between each LiDAR-camera pair ($\mathcal{L}_{\text{sc}}$), and 2) temporal contrastive learning among consecutive LiDAR point clouds across scenes ($\mathcal{L}_{\text{tc}}$).
• Figure 5: Qualitative assessments of state-of-the-art pretraining methods pretrained on nuScenesPanoptic-nuScenes and fine-tuned on nuScenesPanoptic-nuScenes, SemanticKITTISemanticKITTI, and Waymo Opensun2020waymoOpen, with $1\%$ annotations. The error maps show the correct and incorrect predictions in gray and red, respectively. Best viewed in colors and zoomed-in for details.