Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models

TL;DR

OLIVINE tackles self-conflict in image-to-LiDAR contrastive distillation by leveraging Visual Foundation Models to generate weak semantic labels and enforcing semantic-consistent, class-centered representations through a von Mises-Fisher framework. It couples weakly-supervised contrastive distillation with semantic-guided consistency and a density-category aware sampling strategy, addressing both semantic mislabeling in negatives and imbalances in spatial and category distributions. The method achieves state-of-the-art or competitive results on semantic segmentation and 3D object detection across nuScenes and SemanticKITTI, demonstrating robust cross-modal transfer with limited labeled data. Overall, OLIVINE reduces annotation demands while delivering more coherent, discriminative 3D representations for autonomous driving and related tasks, with publicly available code to support reproducibility.

Abstract

Contrastive image-to-LiDAR knowledge transfer, commonly used for learning 3D representations with synchronized images and point clouds, often faces a self-conflict dilemma. This issue arises as contrastive losses unintentionally dissociate features of unmatched points and pixels that share semantic labels, compromising the integrity of learned representations. To overcome this, we harness Visual Foundation Models (VFMs), which have revolutionized the acquisition of pixel-level semantics, to enhance 3D representation learning. Specifically, we utilize off-the-shelf VFMs to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation. Additionally, we employ von Mises-Fisher distributions to structure the feature space, ensuring semantic embeddings within the same class remain consistent across varying inputs. Furthermore, we adapt sampling probabilities of points to address imbalances in spatial distribution and category frequency, promoting comprehensive and balanced learning. Extensive experiments demonstrate that our approach mitigates the challenges posed by traditional methods and consistently surpasses existing image-to-LiDAR contrastive distillation methods in downstream tasks. The source code is available at https://github.com/Eaphan/OLIVINE.

Figures (12)

• Figure 1: The overall pipeline of our proposed OLIVINE. The pipeline starts with feature extraction via a trainable 3D backbone and a pre-trained 2D backbone, followed by feature alignment in a common space. The learning is driven by weakly-supervised contrastive distillation with coarse semantic labels, self-supervised distillation of randomly sampled point-pixel pairs, and semantic consistency regularization through the von Mises-Fisher distribution. Besides, our approach is also characterized by the novel sampling strategy of point-pixel pairs addressing spatial and category distribution imbalances.
• Figure 2: The visual results of various point cloud pretraining strategies, pre-trained on nuScenes and fine-tuned using merely 1% of annotated data, are displayed. To illustrate the distinctions, we mark correctly predicted areas in gray color and incorrect ones in red.
• Figure 3: T-SNE visualization of the extracted point cloud features by PPKT and our OLIVINE (with head $h_{\mathrm{3D}}^{\mathrm{sem}}$). Each feature is colorized based on its ground-truth semantic labels on nuScenes dataset.
• Figure 4: Class distribution at the pixel level for nuScenes dataset.
• Figure 5: Class distribution at the pixel level for SemanticKITTI dataset.