Uni-Gaussians: Unifying Camera and Lidar Simulation with Gaussians for Dynamic Driving Scenarios

TL;DR

This work introduces Uni-Gaussians, a unified Gaussian-based framework for simultaneous camera and LiDAR simulation in dynamic driving scenes. By representing the scene with 2D Gaussian primitives within a Gaussian scene graph and employing rasterization for image rendering alongside Gaussian ray-tracing for LiDAR rendering, it achieves high fidelity while maintaining computational efficiency. The approach is end-to-end differentiable and trained with a multi-term loss across image, depth, intensity, and normal consistency, demonstrating superior quantitative performance over state-of-the-art methods on the Waymo Open Dataset. The combination of a compact 2D Gaussian representation and sensor-tailored rendering pipelines enables scalable and realistic simulation for autonomous driving applications.

Abstract

Ensuring the safety of autonomous vehicles necessitates comprehensive simulation of multi-sensor data, encompassing inputs from both cameras and LiDAR sensors, across various dynamic driving scenarios. Neural rendering techniques, which utilize collected raw sensor data to simulate these dynamic environments, have emerged as a leading methodology. While NeRF-based approaches can uniformly represent scenes for rendering data from both camera and LiDAR, they are hindered by slow rendering speeds due to dense sampling. Conversely, Gaussian Splatting-based methods employ Gaussian primitives for scene representation and achieve rapid rendering through rasterization. However, these rasterization-based techniques struggle to accurately model non-linear optical sensors. This limitation restricts their applicability to sensors beyond pinhole cameras. To address these challenges and enable unified representation of dynamic driving scenarios using Gaussian primitives, this study proposes a novel hybrid approach. Our method utilizes rasterization for rendering image data while employing Gaussian ray-tracing for LiDAR data rendering. Experimental results on public datasets demonstrate that our approach outperforms current state-of-the-art methods. This work presents a unified and efficient solution for realistic simulation of camera and LiDAR data in autonomous driving scenarios using Gaussian primitives, offering significant advancements in both rendering quality and computational efficiency.

Figures (6)

• Figure 1: Point cloud simulation results of the newest SOTA method LiDAR4D and Ours. Our method can accurately simulate various movable entities including both rigid vehicles and non-rigid pedestrians while LiDAR4D fails.
• Figure 2: Method Overview. Gaussians of all elements are defined in their local or canonical spaces, and are deformed and transformed into the world space at a given time $t$. We perform unifying camera and LiDAR simulation for the whole dynamic driving scenarios. For camera image data, we employ rasterization for rendering. For LiDAR data, we compute the intersection between ellipses and rays to construct ray-tracing.
• Figure 3: Qualitative comparison for LiDAR depth and intensity simulation.
• Figure 4: Qualitative comparison for LiDAR point simulation.
• Figure 5: Qualitative comparison the synthesis of novel view with lateral shifts.