Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?

TL;DR

This work tackles the limitations of cross-modal contrastive distillation for 3D representation learning by showing that focusing solely on modality-shared information misses essential modality-specific cues. It introduces CMCR (Cross-Modal Comprehensive Representation Learning), which decouples modality-shared and modality-specific features, employs a multi-modal unified codebook for cross-modal alignment, and uses geometry-enhanced masked image modeling plus occupancy estimation to enrich 3D representations. The framework is supported by theoretical analysis that motivates combining shared information with reconstruction-based signals, and extensive experiments demonstrate superior performance across 3D semantic segmentation, object detection, and panoptic segmentation on diverse datasets, especially in low-label regimes. The approach offers practical benefits for scalable 3D perception and provides a solid foundation for future cross-modal 3D learning research.

Abstract

Cross-modal contrastive distillation has recently been explored for learning effective 3D representations. However, existing methods focus primarily on modality-shared features, neglecting the modality-specific features during the pre-training process, which leads to suboptimal representations. In this paper, we theoretically analyze the limitations of current contrastive methods for 3D representation learning and propose a new framework, namely CMCR (Cross-Modal Comprehensive Representation Learning), to address these shortcomings. Our approach improves upon traditional methods by better integrating both modality-shared and modality-specific features. Specifically, we introduce masked image modeling and occupancy estimation tasks to guide the network in learning more comprehensive modality-specific features. Furthermore, we propose a novel multi-modal unified codebook that learns an embedding space shared across different modalities. Besides, we introduce geometry-enhanced masked image modeling to further boost 3D representation learning. Extensive experiments demonstrate that our method mitigates the challenges faced by traditional approaches and consistently outperforms existing image-to-LiDAR contrastive distillation methods in downstream tasks. Code will be available at https://github.com/Eaphan/CMCR.

Figures (10)

• Figure 1: Performance comparison of our method with scratch training and SLidR sautier2022slidr across multiple benchmarks. A larger covered area indicates superior overall performance.
• Figure 2: Depiction of the mutual information and entropy between the point cloud, image, and task-relevant information.
• Figure 3: The overview of our proposed CMCR. The pipeline integrates both 2D image and 3D point cloud data to learn shared and modality-specific features. The model decouples features into two categories: modality-shared (denoted by $F^{\mathrm{3D}}$ and $F^{\mathrm{2D}}$) and modality-specific (denoted by $G^{\mathrm{3D}}$ and $G^{\mathrm{2D}}$). Contrastive learning is applied to modality-shared features, followed by vector quantization to map them to a unified latent space. The network is driven to learn modality-specific features with masked image restoration and occupancy estimation tasks.
• Figure 4: (a) Illustration of the codebook update process using EMA. (b) Depiction of the commitment loss mechanism, where 2D features are aligned with the codeword selected based on the corresponding 3D features.
• Figure 5: The visual results of different point cloud pretraining methods, where the models were pre-trained on nuScenes and fine-tuned using only 1% of annotated data. Correctly predicted areas are highlighted in gray, while incorrect predictions are marked in red to highlight the differences.

Theorems & Definitions (2)

• theorem 1: Suboptimality of Contrastive Distillation
• theorem 2