Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation

TL;DR

R3DPA addresses the scarcity of large-scale LiDAR data for realistic scene generation by transferring knowledge from RGB image priors and grounding generation in self-supervised 3D representations. It trains in two stages: first aligning a VAE to a pretrained RGB FM to adapt priors to range-image LiDAR data, then performing end-to-end optimization with a 3D representation alignment loss that projects 3D features onto a 2D grid for supervision, enabling a unified latent space for LiDAR synthesis. Key contributions include the first unconditional LiDAR generator that exploits RGB priors and 3D features, state-of-the-art results on KITTI-360, and practical test-time conditioning for object inpainting and scene mixing via the alignment signal $\mathcal{L}_{\textrm{Alignment}}$. This work significantly advances bridging 2D and 3D representations in LiDAR generation, with potential for more controllable and data-efficient 3D scene synthesis in autonomous driving.

Abstract

LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at https://github.com/valeoai/R3DPA.

Figures (8)

• Figure 1: R3DPA pipeline overview. LiDAR point cloud generation from range images commonly follows a two-stage approach: the VAE is trained independently and then frozen, while the generative model is trained on its latent space. In contrast, our method leverages priors from a backbone pretrained on large-scale image datasets. The alignment step trains the VAE from scratch while initializing and freezing the generative model with pretrained weights. This stage ensures that the latent space of our newly trained VAE remains compatible with the knowledge of the pretrained generative model. We then jointly optimize the VAE encoder and the generative model under the supervision of 3D representations. Range VAE denotes a model trained on range images.
• Figure 2: End-to-end training. We jointly train the VAE and the flow matching (FM) model with the additional alignment loss $\mathcal{L}_{\textrm{Alignment}}$, which aligns the internal representations of the FM model with 3D point features $y$. At each step, the VAE is updated with $\mathcal{L}_{\textrm{AE}} + \lambda_1 \mathcal{L}_{\textrm{Alignment}}$, followed by updating the FM model with $\mathcal{L}_{\textrm{Denoising}} + \lambda_2 \mathcal{L}_{\textrm{Alignment}}$. $z_t$ is the interpolated noisy latent vector derived from the clean latent $z$. The input point cloud and its associated features are projected onto the range image $x$ and the feature map $y$, respectively, using an equirectangular projection.
• Figure 3: PCA visualization of 3D features and their projection. The 768-dimensional 3D features puy2024three are reduced to three principal components, which are then mapped to RGB colors for visualization. (Top) Point features in LiDAR scene, (Bottom) Projected features at grid size 8$\times$64, matching the internal representation of the transformer ma_sit_2024.
• Figure 4: Effect of end-to-end training on latents. (Left) Input range images. (Middle) Corresponding PCA visualizations of latents obtained using the standard VAE paradigm, such as ran_towards_2024, where the VAE is trained independently on range images. (Right) PCA visualizations of latents from our end-to-end training with 3D representation alignment. Latents obtained through end-to-end training are more expressive, allowing a clear distinction of objects and background in the scene.
• Figure 5: Qualitative comparison of unconditional generations. Our model generates high-quality point clouds with diverse and realistic objects.