LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation

TL;DR

LiDAR-RT tackles the challenge of real-time, physics-aware LiDAR re-simulation in dynamic driving scenes by introducing Gaussian primitives with learnable LiDAR properties, scene graphs for dynamics, and a differentiable Gaussian-based ray tracer accelerated by BVH and NVIDIA OptiX. The approach enables high-fidelity range-image rendering at real-time frame rates and supports flexible scene editing and sensor configurations, outperforming state-of-the-art NeRF-based and hybrid methods in both quality and efficiency. Key contributions include a differentiable rendering pipeline that maps Gaussian primitives to LiDAR radiance, a robust optimization framework with depth/intensity/ray-drop/Chamfer-distance losses, and a UNet-based refinement for sensor-level ray-drop. The results on Waymo Open and KITTI-360 demonstrate strong rendering accuracy, fast performance, and practical applicability to downstream tasks, with limitations noted for non-rigid objects and long driving sequences, guiding future work toward non-rigid modeling and scalability enhancements.

Abstract

This paper targets the challenge of real-time LiDAR re-simulation in dynamic driving scenarios. Recent approaches utilize neural radiance fields combined with the physical modeling of LiDAR sensors to achieve high-fidelity re-simulation results. Unfortunately, these methods face limitations due to high computational demands in large-scale scenes and cannot perform real-time LiDAR rendering. To overcome these constraints, we propose LiDAR-RT, a novel framework that supports real-time, physically accurate LiDAR re-simulation for driving scenes. Our primary contribution is the development of an efficient and effective rendering pipeline, which integrates Gaussian primitives and hardware-accelerated ray tracing technology. Specifically, we model the physical properties of LiDAR sensors using Gaussian primitives with learnable parameters and incorporate scene graphs to handle scene dynamics. Building upon this scene representation, our framework first constructs a bounding volume hierarchy (BVH), then casts rays for each pixel and generates novel LiDAR views through a differentiable rendering algorithm. Importantly, our framework supports realistic rendering with flexible scene editing operations and various sensor configurations. Extensive experiments across multiple public benchmarks demonstrate that our method outperforms state-of-the-art methods in terms of rendering quality and efficiency. Our project page is at https://zju3dv.github.io/lidar-rt.

Figures (12)

• Figure 1: Realistic and real-time rendering of LiDAR view in dynamic driving scenes. Our LiDAR-RT produces high-fidelity LiDAR view at 30 FPS (64×2650) within 2 hours of training. SOTA method zheng2024lidar4d struggles to model the dynamic objects in complex scenes and suffers from high training and rendering costs (15 hours for training and 0.2 FPS for rendering a range image).
• Figure 2: Overview of LiDAR-RT. (a) We decompose the dynamic scene into a background model and multiple object models, with each represented by a set of Gaussian primitives. In addition to geometric attributes, we introduce learnable parameters (SHs) on Gaussians to emulate the intrinsic properties $(\zeta, \beta)$ of LiDAR sensors. (b) Based on this representation, we design a differentiable ray tracing framework. We first construct the proxy geometry for Gaussian primitives and then cast rays from the sensor to perform intersection tests. (c) By evaluating the response from these intersections, we accumulate point-wise properties along each ray, and finally render the novel LiDAR view as range images. (d) The range images are fused and re-projected into LiDAR point clouds for downstream tasks.
• Figure 3: Ray intersection with proxy geometries. We construct the proxy geometry for each 2D Gaussian primitive as a pair of co-planar triangles, then the ray tracer performs intersection tests with the vertices $\vec{v}$ and surface normal $\vec{n}$.
• Figure 4: Qualitative comparison on the KITTI-360 liao2022kitti dataset. Dynamic vehicles are zoomed in for better visualization.
• Figure 5: Scene editing results on the Waymo dataset sun2020waymo. Our method supports various scene editing operations, including trajectory manipulation, object insertion, and object removal. The boxes ($\textcolor{cyan}{\Box} / \textcolor{pink}{\Box}$) highlight the edited objects.