GS-LIVM: Real-Time Photo-Realistic LiDAR-Inertial-Visual Mapping with Gaussian Splatting

TL;DR

GS-LIVM is introduced, a real-time photo-realistic LiDAR-Inertial-Visual mapping framework with Gaussian Splatting tailored for outdoor scenes that achieves state-of-the-art performance in terms of mapping efficiency and rendering quality.

Abstract

In this paper, we introduce GS-LIVM, a real-time photo-realistic LiDAR-Inertial-Visual mapping framework with Gaussian Splatting tailored for outdoor scenes. Compared to existing methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), our approach enables real-time photo-realistic mapping while ensuring high-quality image rendering in large-scale unbounded outdoor environments. In this work, Gaussian Process Regression (GPR) is employed to mitigate the issues resulting from sparse and unevenly distributed LiDAR observations. The voxel-based 3D Gaussians map representation facilitates real-time dense mapping in large outdoor environments with acceleration governed by custom CUDA kernels. Moreover, the overall framework is designed in a covariance-centered manner, where the estimated covariance is used to initialize the scale and rotation of 3D Gaussians, as well as update the parameters of the GPR. We evaluate our algorithm on several outdoor datasets, and the results demonstrate that our method achieves state-of-the-art performance in terms of mapping efficiency and rendering quality. The source code is available on GitHub.

Figures (13)

• Figure 1: Illustration of our photo-realistic mapping results on Botanic Garden Sequence 1018_13. The pink dashed line represents the running trajectory, with the entire trajectory being approximately 200 meters. The green square is the starting point and the red pentagram is the endpoint. The rendering images at different locations are shown in the corners. The quantitative metrics are shown in the table.
• Figure 2: The system processes input from point cloud data obtained from LiDAR, motion data from an IMU, and monocular color image captured by a camera. In the tracking thread, the ESIKF algorithm is employed to track motion, producing odometry output at the IMU frequency. In the mapping thread, the rendered color point cloud is utilized for Voxel-GPR, after which the initialized 3D Gaussian data is integrated into a dense 3D Gaussian map for rendering optimization. The final output is a high-quality, dense 3D Gaussian map. Notations C, D, and S denote the rasterized color image, depth image, and silhouette image, respectively. $\oplus$ represents the global hash-colored map, which provides neighboring query points for the recently scanned LiDAR points.
• Figure 3: (a) Illustration of Voxel-GPR for the $\alpha$th voxel involves processing voxels that contain a sufficient number $\tau$ of points. The depicted surface illustrates the distribution of the predicted points, where the variance decreases from areas marked in red to those in green. (b) Illustration depicts the initialization of 3D Gaussians. As indicated by the curved orange arrow, we consider the points within the neighborhood (black dash region). The green spheres represent the fitted 3D Gaussians.
• Figure 4: This illustration presents four types of voxels in hash voxel map. Small dots marked in black represent new scanned points cloud from LiDAR, with their variance corresponding to the sensor's inherent noise. Larger dots highlighted in dark blue are from the meshgrid, poised for processing via Voxel-GPR, while the small dots in light blue have undergone variance updates. The ellipsoids are shaded in various colors, where each color signifies the magnitude of their variance.
• Figure 5: Illustration of the rendering performance on the NeRF-based method rosinolnerf2023 and the 3DGS-based method matsuki2024gaussiankerbl20233d on sequence hkust_campus_seq_00, Visual_Challenge, eee_03, 1005_00, respectively.