RaLD: Generating High-Resolution 3D Radar Point Clouds with Latent Diffusion

TL;DR

RaLD tackles the challenge of sparse, low-resolution radar point clouds by generating dense LiDAR-like 3D scenes from raw radar spectra. It introduces a latent-diffusion framework operating in a compact latent space learned by a frustum-based LiDAR autoencoder, augmented with order-invariant latent encoding and radar spectrum guidance to condition generation. The approach yields high-fidelity 3D reconstructions and surpasses state-of-the-art baselines on ColoRadar, with ablations validating the contributions of radar conditioning, query initialization, and frustum-aligned occupancy. This work enables robust, high-resolution 3D perception from radar in challenging environments and demonstrates scalable, transferable performance across indoor and outdoor scenarios.

Abstract

Millimeter-wave radar offers a promising sensing modality for autonomous systems thanks to its robustness in adverse conditions and low cost. However, its utility is significantly limited by the sparsity and low resolution of radar point clouds, which poses challenges for tasks requiring dense and accurate 3D perception. Despite that recent efforts have shown great potential by exploring generative approaches to address this issue, they often rely on dense voxel representations that are inefficient and struggle to preserve structural detail. To fill this gap, we make the key observation that latent diffusion models (LDMs), though successful in other modalities, have not been effectively leveraged for radar-based 3D generation due to a lack of compatible representations and conditioning strategies. We introduce RaLD, a framework that bridges this gap by integrating scene-level frustum-based LiDAR autoencoding, order-invariant latent representations, and direct radar spectrum conditioning. These insights lead to a more compact and expressive generation process. Experiments show that RaLD produces dense and accurate 3D point clouds from raw radar spectrums, offering a promising solution for robust perception in challenging environments.

Figures (7)

• Figure 1: RaLD uses latent diffusion with sparse vectors, enabling high-fidelity 3D radar point cloud generation beyond dense-grid methods.
• Figure 2:
• Figure 3: Frustum-based occupancy partitioning aligned with the capture geometry of LiDAR and radar. Volumetric cells are defined in polar coordinates—range, azimuth, and elevation—to match the sensors’ angular sampling patterns.
• Figure 4: Order-sensitive (left) vs. order-invariant (right) latent encodings. Our method ensures consistent training targets despite the order of input point cloud.
• Figure 5: Visualization of 3D radar point cloud generation result.