CLAP: Unsupervised 3D Representation Learning for Fusion 3D Perception via Curvature Sampling and Prototype Learning

TL;DR

CLAP tackles the challenge of label-free, joint unsupervised pre-training for fusion 3D perception by combining curvature-guided sampling with neural-field differentiable rendering and a cross-modal prototype learning framework. An EM-based training scheme aligns image and LiDAR embeddings to a shared set of learnable prototypes, while a swapping prototype prediction loss and Gram Matrix Regularization stabilize training and encourage cross-modal interaction. The method pre-trains LiDAR, camera, and fusion backbones end-to-end, yielding substantial gains on NuScenes and Waymo in few-shot settings and demonstrating strong scaling with reduced fine-tuning data. Overall, CLAP advances label-efficient multimodal 3D perception by exploiting curvature-informed sampling and cross-modal semantics and geometry during pre-training.

Abstract

Unsupervised 3D representation learning reduces the burden of labeling multimodal 3D data for fusion perception tasks. Among different pre-training paradigms, differentiable-rendering-based methods have shown most promise. However, existing works separately conduct pre-training for each modalities due to computational costs of processing large point clouds with images. As such, mutual benefit of high-level semantics (from image) and 3D structure (from point cloud) has not been exploited. To address this gap, we propose a joint unsupervised differentiable-rendering-based pre-training method for images and point clouds, termed CLAP, short for Curvature sampLing and leArnable Prototype. Specifically, our method overcomes the computational hurdle by Curvature Sampling to select the more informative points/pixels for pre-training. To uncover the performance benefits brought by their complementarity, we propose to use learnable prototypes to represent parts of the 3D scenes in a common feature space and an Expectation-Maximization training scheme to associate embeddings of each modality to prototypes. We further propose a swapping prediction loss that explores their interplay through prototypes along with a Gram Matrix Regularization term to maintain training stability. Experiments on NuScenes and Waymo datasets show that CLAP achieves up to 100% more performance gain as compared to previous SOTA pre-training methods. Codes and models will be released.

Figures (4)

• Figure 1: Unlike previous SOTA unsupervised 3D representation learning method UniPAD unipad that separately pre-train LiDAR and camera encoders with differential rendering (a), our proposed method CLAP conducts joint pre-training for fusion perception.
• Figure 2: The pipeline of CLAP. In order to jointly pre-train the LiDAR, camera and fusion encoders, we first embed the paired LiDAR point clouds and camera images with $f^{\text{enc}}_{\text{P}}$, $f^{\text{enc}}_{\text{I}}$ and $f^{\text{enc}}_{\text{fusion}}$. Then based on the fusion features, CLAP applies differentiable rendering to predict both depth and rgb with the SDF and RGB values of the sampled points along LiDAR/camera rays from $f^{\text{SDF}}$ and $f^{\text{RGB}}$, with which we compute loss against the observed LiDAR point cloud and camera images. To make joint pre-training feasible, we propose Curvature Sampling to sample informative parts of the 3D scene, as descriped in (a). Furthermore, we propose to use learnable prototypes to represent parts of objects in a common feature space and utilize an Expectation-Maximization approach to maximize the similarity between prototypes and 3D embeddings of each modality. To delve deeper into the interplay of image semantics and LiDAR geometry, we use swapping prototype prediction loss. Finally, we propose a Gram Matrix Regularization loss to prevent collapse of prototype learning.
• Figure 3: Inspiration of Curvature Sampling. It can be found that areas with low curvature tend to be less informative, like road plane, while those with high curvature provide more information (objects like cars).
• Figure 4: Visualization of curvature estimation. The color of the points change from blue to red, indicating lower sample weight / curvature to higher ones. We highlight those correct regions with orange boxes and those noisy estimation with green ones. Some of the correct estimated regions are further zoom-in for better understanding. Best view in color.