LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving

TL;DR

LargeAD tackles the challenge of cross-sensor 3D pretraining for autonomous driving by distilling semantic knowledge from Vision Foundation Models into LiDAR-based representations. It introduces a VFM-driven semantic superpixel framework, a cross-modal contrastive objective, and temporal/spatial consistency constraints, enhanced by multi-source data pretraining to generalize across diverse LiDAR sensors. The approach yields consistent gains on LiDAR segmentation and 3D object detection across 11 datasets, including robustness tests and zero-shot scenarios. This work provides a scalable, robust pathway for leveraging 2D foundation-models to improve 3D perception in real-world driving systems.

Abstract

Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: (i) VFM-driven superpixel generation for detailed semantic representation, (ii) a VFM-assisted contrastive learning strategy to align multimodal features, (iii) superpoint temporal consistency to maintain stable representations across time, and (iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach achieves substantial gains over state-of-the-art methods in linear probing and fine-tuning for LiDAR-based segmentation and object detection. Extensive experiments on 11 large-scale multi-sensor datasets highlight our superior performance, demonstrating adaptability, efficiency, and robustness in real-world autonomous driving scenarios.

Figures (9)

• Figure 1: Comparisons between i) the conventional image-to-LiDAR data pretraining frameworks sautier2022slidrmahmoud2023stliu2023seal and ii) our proposed large-scale cross-sensor data pretraining (LargeAD). Our approach combines heterogeneous data sources for representation learning, which achieves superior robustness and scalability. Different from previous work, our framework encourages representation learning across different datasets, which largely enhances the generalizability.
• Figure 2: Illustration of image-to-LiDAR data pretraining using i) the heuristic SLIC algorithm achanta2012slic and ii) different vision foundation models (VFMs). Images in the first row are the superpixels generated by different methods, where each color represents one distinct segment. The LiDAR point clouds from the second row are the superpoints grouped by projecting superpixels to 3D using camera-LiDAR correspondence. The third row shows the linear probing results after data pretraining.
• Figure 3: Overview of the VFM-driven image-to-LiDAR contrastive learning framework. Given a pair of LiDAR point cloud $\mathcal{P}^{t}$ and camera image $\mathcal{I}^{t}$ captured at timestamp $t$, along with another LiDAR point cloud $\mathcal{P}^{t+n}$ captured at timestamp $t+n$, we generate semantic superpixels using vision foundation models (VFMs). The corresponding superpoints are obtained by projecting image pixels onto the point cloud. Two key objectives are established: i) spatial contrastive learning between paired LiDAR and camera features, and ii) temporal consistency regularization between point segments from $\mathcal{P}^{t}$ and $\mathcal{P}^{t+n}$.
• Figure 4: The positive feature correspondences of the contrastive objective in our proposed VFM-driven contrastive learning framework. The circles and triangles represent the instance-level and the point-level features, respectively.
• Figure 5: The cosine similarity between a query point (denoted as the red dot) and the feature learned with SLIC achanta2012slic and different VFMs kirillov2023samzou2023xcoderzhang2023openSeeDzou2023seem. The queried semantic classes from top to bottom examples are: "car", "manmade", and "truck". The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in colors.