GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting

TL;DR

GS-LiDAR tackles LiDAR novel-view synthesis in driving scenarios by replacing slow NeRF-based representations with panoramic Gaussian splatting using 2D Gaussian primitives and periodic vibration to model static and dynamic surfaces. A panoramic ray-splat rendering pipeline produces depth, intensity, and ray-drop maps, supervised by ground-truth LiDAR data and refined by a U-Net; the method optimizes with depth distortion, normal consistency, and Chamfer losses to align rendered and real data. It demonstrates superior performance on KITTI-360 and nuScenes, with notable training speedups ($\sim$1.67×) and rendering speedups (up to $\sim$31×) over LiDAR4D, along with improved depth and intensity metrics and more cohesive point clouds. The approach offers a scalable, efficient LiDAR NVS framework that better handles dynamic driving environments, enabling faster, more realistic LiDAR data generation for autonomous driving research and development.

Abstract

LiDAR novel view synthesis (NVS) has emerged as a novel task within LiDAR simulation, offering valuable simulated point cloud data from novel viewpoints to aid in autonomous driving systems. However, existing LiDAR NVS methods typically rely on neural radiance fields (NeRF) as their 3D representation, which incurs significant computational costs in both training and rendering. Moreover, NeRF and its variants are designed for symmetrical scenes, making them ill-suited for driving scenarios. To address these challenges, we propose GS-LiDAR, a novel framework for generating realistic LiDAR point clouds with panoramic Gaussian splatting. Our approach employs 2D Gaussian primitives with periodic vibration properties, allowing for precise geometric reconstruction of both static and dynamic elements in driving scenarios. We further introduce a novel panoramic rendering technique with explicit ray-splat intersection, guided by panoramic LiDAR supervision. By incorporating intensity and ray-drop spherical harmonic (SH) coefficients into the Gaussian primitives, we enhance the realism of the rendered point clouds. Extensive experiments on KITTI-360 and nuScenes demonstrate the superiority of our method in terms of quantitative metrics, visual quality, as well as training and rendering efficiency.

Figures (11)

• Figure 1: GS-LiDAR achieves superior LiDAR simulation quality for novel view synthesis while maintaining fast training and rendering speed.
• Figure 2: Overview of the GS-LiDAR framework: GS-LiDAR is based on 2D Gaussian primitives with periodic vibration properties, allowing for dynamic modeling of position and opacity along with accurate geometry. At a given timestamp, Gaussians query their states and utilize the proposed panoramic Gaussian splatting technique to render panoramic maps of depth, ray-drop, and intensity. For each ray and Gaussian primitive, we calculate their intersection to obtain the depth and $\alpha$, ensuring more geometrically consistent renderings. The results are subsequently refined by a well-trained U-Net to further enhance the quality of the point clouds.
• Figure 3: Our LiDAR coordinate system and two ways of depth rendering. The mean depth refers to the weighted average of each depth using the rendering weights, while the median depth is defined as the maximum depth with transparency, i.e., $\prod_{j=1}^{k-1} (1 - o_j\,\mathcal{G}_j)$, no greater than 0.5.
• Figure 4: Panoramic Gaussian rasterization details. (a) Our method employs tile-based sorting and rendering. For panoramic ray maps, we first transform the epipolar coordinate system into the pixel coordinate system. The pixel map is then divided into small tiles, and within each tile, Gaussian primitives are sorted based on their distance to the LiDAR origin. (b) During pixel rendering, the $\alpha$ and depth are computed by calculating the intersection between the ray and the Gaussian primitive.
• Figure 5: Comparison of 3D LiDAR point cloud. GS-LiDAR produces a more cohesive LiDAR point cloud compared to LiDAR-NeRF tao2023lidarnerf and LiDAR4D zheng2024lidar4d.