Dynamic LiDAR Re-simulation using Compositional Neural Fields

TL;DR

DyNFL presents a compositional neural-field framework to re-simulate LiDAR scans in dynamic driving scenes by decomposing a scene into a static background and multiple dynamic neural fields, one per moving vehicle. It introduces a ray-drop-based neural field composition to accurately handle occlusions and transparent surfaces when aggregating measurements from several neural assets. The approach relies on an SDF-based volume rendering formulation to enforce geometry fidelity and a two-stage rendering pipeline that selectively evaluates only the fields likely intersected by a given ray, improving efficiency. Extensive experiments on real and synthetic data demonstrate improved range and intensity fidelity, better alignment with downstream perception tasks, and powerful scene editing capabilities such as inserting, removing, or repositioning assets. The work advances counterfactual LiDAR re-simulation for autonomous driving by balancing physical realism with flexible content editing.

Abstract

We introduce DyNFL, a novel neural field-based approach for high-fidelity re-simulation of LiDAR scans in dynamic driving scenes. DyNFL processes LiDAR measurements from dynamic environments, accompanied by bounding boxes of moving objects, to construct an editable neural field. This field, comprising separately reconstructed static background and dynamic objects, allows users to modify viewpoints, adjust object positions, and seamlessly add or remove objects in the re-simulated scene. A key innovation of our method is the neural field composition technique, which effectively integrates reconstructed neural assets from various scenes through a ray drop test, accounting for occlusions and transparent surfaces. Our evaluation with both synthetic and real-world environments demonstrates that DyNFL substantially improves dynamic scene LiDAR simulation, offering a combination of physical fidelity and flexible editing capabilities.

Figures (13)

• Figure 1: Overview of DyNFL. Our method takes LiDAR scans and tracked bounding boxes of dynamic vehicles as input. DyNFL first decomposes the scene into a static background and $N$ dynamic vehicles, each modelled using a dedicated neural field. These neural fields are then composed to re-simulate LiDAR scans in dynamic scenes. Our composition technique supports various scene edits, including altering object trajectories, removing and adding reconstructed neural assets between scenes.
• Figure 2: Qualitative comparison of range estimation on Waymo Dynamic dataset. Dynamic vehicles are zoomed in, and points are color-coded by range errors (-100 100 cm).
• Figure 3: ECDF plots showcasing range errors across all the points (left) and specifically for points on dynamic vehicles (right). Our composition of neural fields outperforms LiDARsim manivasagam2020lidarsim and UniSim yang2023unisim, especially when it comes to dynamic vehicles.
• Figure 4: Qualitative results of range estimation. Regions with gross errors (-100 100 cm) are highlighted.
• Figure 5: Qualitative results on Waymo Dynamic dataset. Our model equipped with a ray drop module effectively composites multiple neural fields, re-simulating LiDAR scans of high quality.