NeuRadar: Neural Radiance Fields for Automotive Radar Point Clouds

TL;DR

NeuRadar addresses the lack of NeRF-based automotive radar modeling by extending a unified neural feature field to jointly render radar point clouds alongside camera and lidar data. It introduces a data-driven radar decoder that leverages NeRF-derived depth and a Transformer to predict radar detections in either a deterministic or probabilistic MB-RFS form, enabling realistic novel-view radar synthesis. The method is validated on VoD and an extended Zenseact Open Dataset (ZOD) with radar data, showing improved radar reconstruction over baselines, particularly at longer ranges, and supporting novel scenario generation. The authors release radar data for ZOD, together with open-source code, establishing a baseline and evaluation framework for NeRF-based radar simulation that can catalyze multi-modal radar research in autonomous driving.

Abstract

Radar is an important sensor for autonomous driving (AD) systems due to its robustness to adverse weather and different lighting conditions. Novel view synthesis using neural radiance fields (NeRFs) has recently received considerable attention in AD due to its potential to enable efficient testing and validation but remains unexplored for radar point clouds. In this paper, we present NeuRadar, a NeRF-based model that jointly generates radar point clouds, camera images, and lidar point clouds. We explore set-based object detection methods such as DETR, and propose an encoder-based solution grounded in the NeRF geometry for improved generalizability. We propose both a deterministic and a probabilistic point cloud representation to accurately model the radar behavior, with the latter being able to capture radar's stochastic behavior. We achieve realistic reconstruction results for two automotive datasets, establishing a baseline for NeRF-based radar point cloud simulation models. In addition, we release radar data for ZOD's Sequences and Drives to enable further research in this field. To encourage further development of radar NeRFs, we release the source code for NeuRadar.

Figures (7)

• Figure 1: NeuRadar generates radar point clouds alongside camera and lidar data for novel viewpoints and altered scenes. This figure illustrates images, radar point clouds (in red), and lidar point clouds (in blue) generated by NeuRadar for two sequences in ZOD.
• Figure 2: NeuRadar: our multimodal novel view rendering method for autonomous driving. (a) Rays from each sensor modality render ray features from the NFF. Camera and lidar branches decode their ray features into RGB values via an upsampling CNN and into lidar ray drop probability and intensity via MLPs, respectively. Radar ray features, along with estimated depths from the NFF, generate a radar point cloud via a specialized radar decoder. (b) The ray return position $({x}_{\text{NFF}}, {y}_{\text{NFF}}, {z}_{\text{NFF}})$ is obtained from the known azimuth, elevation, and estimated depth, while a Transformer predicts offsets $(\delta_x, \delta_y, \delta_z)$ and a detection confidence score $r$. The sum of the ray return position and offsets determines the final position. As an extension, the probabilistic method uses a Laplace parameter head to predict the scaling parameters. The set $\{r,\, {x}_{\text{NFF}},\, \delta_{x},\, b_{x},\, {y}_{\text{NFF}},\, \delta_{y},\, b_{y},\, {z}_{\text{NFF}},\, \delta_{z},\, b_{z}\}$ provides all necessary information for modeling the MB RFS.
• Figure 3: Illustration of our design, where radar-projected rays cover the full sensor field of view (FOV). The virtual grid (grey dashed lines) indicates the complete FOV, with blue points denoting uniformly sampled locations. The direction from the sensor to each blue point defines a ray.
• Figure 4: Four sampled sets (left) drawn from a Bernoulli RFS (right) with an existence probability of $r=0.75$. The RFS accounts for both the absence of objects and spatial uncertainties. The image is provided for context only.
• Figure 5: Novel view synthesis for interpolated frames including radar (red) and lidar (blue) detections. The Deterministic (Det.) and Probabilistic (Prob.) methods outperform the baseline at larger distances.