NeuRAD: Neural Rendering for Autonomous Driving

TL;DR

NeuRAD tackles the challenge of scalable, sensor realistic neural rendering for autonomous driving by introducing a single unified neural feature field that models both static and dynamic scene elements with time awareness and actor level encoding. The approach integrates comprehensive sensor modeling, including rolling shutter, beam divergence, and lidar ray dropping, and employs efficient two-stage ray sampling with per-sensor appearance embeddings to render realistic 360-degree camera and lidar data. Empirically, NeuRAD achieves state-of-the-art performance across five major AD datasets for both novel view synthesis and lidar-based metrics, while offering practical editing capabilities such as altering actor trajectories and ego poses for closed-loop simulation and data augmentation. The authors provide extensive ablations and demonstrate extrapolation capabilities, and they release the code to support community development and benchmarking.

Abstract

Neural radiance fields (NeRFs) have gained popularity in the autonomous driving (AD) community. Recent methods show NeRFs' potential for closed-loop simulation, enabling testing of AD systems, and as an advanced training data augmentation technique. However, existing methods often require long training times, dense semantic supervision, or lack generalizability. This, in turn, hinders the application of NeRFs for AD at scale. In this paper, we propose NeuRAD, a robust novel view synthesis method tailored to dynamic AD data. Our method features simple network design, extensive sensor modeling for both camera and lidar -- including rolling shutter, beam divergence and ray dropping -- and is applicable to multiple datasets out of the box. We verify its performance on five popular AD datasets, achieving state-of-the-art performance across the board. To encourage further development, we will openly release the NeuRAD source code. See https://github.com/georghess/NeuRAD .

Figures (13)

• Figure 1: NeuRAD is a neural rendering method tailored to dynamic automotive scenes. With it, we can alter the pose of the ego vehicle and other road users as well as freely add and/or remove actors. These capabilities make NeuRAD suitable to serve as the foundation in components such as sensor-realistic closed-loop simulators or powerful data augmentation engines.
• Figure 2: Overview of our approach. We learn a joint neural feature field for the statics and dynamics of an automotive scene, where the two are discerned only by our actor-aware hash encoding. Points that fall inside actor bounding boxes are transformed to actor-local coordinates and, together with actor index, used to query the 4D hash grid. We decode the volume rendered ray-level features to RGB values using an upsampling CNN, and to ray drop probability and intensity using MLPs.
• Figure 3: Impact of modeling rolling shutter in a high-speed scenario (with inset PSNR). (a) original side-camera image. Omitting the rolling shutter entirely (b) results in extremely blurry renderings and unrealistic geometry, especially for the pole. Modeling the lidar rolling shutter (c) improves the quality, but it is only when both sensors are modeled correctly (d) that we get realistic renderings.
• Figure 4: Visualization of ray drop effects for lidar simulation. Highlighted parts show areas where ray dropping effects are important to consider in order to simulate realistic point clouds. CD denotes Chamfer distance normalized by num. GT points.
• Figure 5: Lidar scans in spherical coordinates at different stages during inference of missing lidar rays. The color indicates range, where missing points have been set to a large distance for visualization purposes. Note that we do not add missing points for the two bottom rows, as they typically hit the ego vehicle.