Radar Fields: Frequency-Space Neural Scene Representations for FMCW Radar

TL;DR

The approach unites an explicit, physics-informed sensor model with an implicit neural geometry and reflectance model to directly synthesize raw radar measurements and extract scene occupancy and validate the method’s effectiveness across diverse outdoor scenarios, including urban scenes with dense vehicles and infrastructure, and harsh weather scenarios, where mm-wavelength sensing is favorable.

Abstract

Neural fields have been broadly investigated as scene representations for the reproduction and novel generation of diverse outdoor scenes, including those autonomous vehicles and robots must handle. While successful approaches for RGB and LiDAR data exist, neural reconstruction methods for radar as a sensing modality have been largely unexplored. Operating at millimeter wavelengths, radar sensors are robust to scattering in fog and rain, and, as such, offer a complementary modality to active and passive optical sensing techniques. Moreover, existing radar sensors are highly cost-effective and deployed broadly in robots and vehicles that operate outdoors. We introduce Radar Fields - a neural scene reconstruction method designed for active radar imagers. Our approach unites an explicit, physics-informed sensor model with an implicit neural geometry and reflectance model to directly synthesize raw radar measurements and extract scene occupancy. The proposed method does not rely on volume rendering. Instead, we learn fields in Fourier frequency space, supervised with raw radar data. We validate the effectiveness of the method across diverse outdoor scenarios, including urban scenes with dense vehicles and infrastructure, and in harsh weather scenarios, where mm-wavelength sensing is especially favorable.

Figures (5)

• Figure 1: Radar Fields recovers a 3D scene from raw FFTs of 2D bird’s eye view radar scans. Each radar frame captures information centered around a 2D circular disk (bottom-left), from which a volume-rendering-free representation of the scene is learned (right). Following the antenna gain profile, points along super-sampled azimuthal rays are converted by $f_\chi$ into an embedding. This embedding is subsequently processed by $f_\alpha$ and $f_{\rho\gamma}$, which decompose the signal intensity into occupancy $\alpha$ and reflectance $\rho\gamma$. To reconstruct FFT measurements, we integrate the super-sampled returns from both representations for each range-azimuth point, and apply our forward model (top-right).
• Figure 2: Multi-modal Dataset for Training and Validation. The waterproof sensor rig (left) which is used to collect our dataset and 5 exemplary scenes (right) with (1st row) 40-meter-radius BEV radar returns with the car in the center and pointing to the right, and (2nd row) point clouds from our forward-facing LiDAR, color-coded by height, with the car in the bottom-left and (3rd row) images from our forward-facing RGB camera.
• Figure 3: Radar Fields for Novel Radar View Synthesis. Radar Fields is capable of synthesizing high-quality raw radar FFT measurements at novel viewpoints. Without our proposed super-sampling procedure, predicted measurements become noise prone and inaccurate in magnitude.
• Figure 4: Radar Fields for Scene Reconstruction. Conventional post-processed radar point clouds (second row) are sparse, and, hence, conventional grid mapping methods grid_mapping (third row) fail to recover accurate geometry. Radar Fields relies on raw frequency-space radar measurements and recovers high-quality BEV occupancy (fourth row), and even accurate 3D geometry (third row) from the same 2D radar scans. Without physics-based ray importance sampling (last row), the predicted occupancy becomes inconsistent.
• Figure 5: Radar Fields for adverse weather. Our method is robust to extreme weather and lighting conditions, including low-light and foggy scenes. RGB captures are reported in the top row. We compare our method to LiDAR-NeRF tao2023lidarnerf for LiDAR and Instant-NGP muller2022instantNGP for camera inputs as alternative modalities. LiDAR reconstructions struggle to capture accurate scene geometry due to backscatter. Multi-view reconstruction via RGB fails for this monocular foggy trajectory, also indicated by the synthesized RGB frames (second to last row). Radar Fields exhibits minimal degradation compared to good weather conditions.