LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction

TL;DR

This work proposes a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering and is the first to focus on the more challenging and highly relevant highway scenes for autonomous driving.

Abstract

Photorealistic 3D scene reconstruction plays an important role in autonomous driving, enabling the generation of novel data from existing datasets to simulate safety-critical scenarios and expand training data without additional acquisition costs. Gaussian Splatting (GS) facilitates real-time, photorealistic rendering with an explicit 3D Gaussian representation of the scene, providing faster processing and more intuitive scene editing than the implicit Neural Radiance Fields (NeRFs). While extensive GS research has yielded promising advancements in autonomous driving applications, they overlook two critical aspects: First, existing methods mainly focus on low-speed and feature-rich urban scenes and ignore the fact that highway scenarios play a significant role in autonomous driving. Second, while LiDARs are commonplace in autonomous driving platforms, existing methods learn primarily from images and use LiDAR only for initial estimates or without precise sensor modeling, thus missing out on leveraging the rich depth information LiDAR offers and limiting the ability to synthesize LiDAR data. In this paper, we propose a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering. Unlike prior works that are tested mostly on urban datasets, to the best of our knowledge, we are the first to focus on the more challenging and highly relevant highway scenes for autonomous driving, with sparse sensor views and monotone backgrounds. Visit our project page at: https://umautobots.github.io/lihi_gs

Figures (13)

• Figure 1: LiHi-GS (second row) provides higher quality color and depth renderings for interpolated novel views and for ego/actor shifts compared to state-of-the-art NeRF and GS-based methods neuradstreetgs LiHi-GS does particularly well on actor shifts at longer-ranges (205 meters).
• Figure 2: (Left) Issues with LiDAR projected pseudo-depth supervision, highlighted in the orange box. Points from both near and far distances can map to the same image pixel, resulting in depth ambiguity. In the rendered opacity view (middle), vehicles appear distorted in the case of pseudo-depth supervision, whereas LiHi-GS (right) preserves object geometry and integrity.
• Figure 3: System overview. LiHi-GS takes multiple cameras, LiDAR, and annotated 3D poses as input. During preprocessing, a LiDAR map combined with a COLMAP sparse point cloud is used to initialize the static scene, while LiDAR points aggregated with human-annotated bounding boxes are used to initialize dynamic objects. During training, images and LiDAR scans are rendered separately and compared with the corresponding ground truth images and LiDAR scans for scene reconstruction.
• Figure 4: Depth uncertainty filter removes floating artifacts on object edges from the rendered point cloud.
• Figure 5: LiDAR depth rendering can lead to incorrect depth estimates at object edges, causing floating artifacts between adjacent objects. To address this, depth uncertainty is used to filter out the artifact on object edges, as shown in Figure \ref{['fig:depth uncertainty point cloud']}.