Robust LiDAR-Camera Calibration with 2D Gaussian Splatting

TL;DR

The paper tackles the challenge of targetless LiDAR–camera extrinsic calibration by exploiting 2D Gaussian Splatting (2DGS) to build a differentiable geometric representation from LiDAR frames. It separates geometry construction from colorization, first optimizing 2DGS geometry using depth supervision, then jointly refining splat colors and the LiDAR–camera extrinsic, guided by photometric, reprojection, and triangulation losses. To address the limitations of photometric optimization, it introduces depth uncertainty weighting and two additional geometric losses to enforce cross-view consistency and depth accuracy. Evaluations on KITTI show improved calibration accuracy and robustness to large initial miscalibrations compared to several baselines, highlighting practical applicability for autonomous systems with minimal manual setup.

Abstract

LiDAR-camera systems have become increasingly popular in robotics recently. A critical and initial step in integrating the LiDAR and camera data is the calibration of the LiDAR-camera system. Most existing calibration methods rely on auxiliary target objects, which often involve complex manual operations, whereas targetless methods have yet to achieve practical effectiveness. Recognizing that 2D Gaussian Splatting (2DGS) can reconstruct geometric information from camera image sequences, we propose a calibration method that estimates LiDAR-camera extrinsic parameters using geometric constraints. The proposed method begins by reconstructing colorless 2DGS using LiDAR point clouds. Subsequently, we update the colors of the Gaussian splats by minimizing the photometric loss. The extrinsic parameters are optimized during this process. Additionally, we address the limitations of the photometric loss by incorporating the reprojection and triangulation losses, thereby enhancing the calibration robustness and accuracy.

Figures (6)

• Figure 1: Overview of the proposed method. The proposed method uses LiDAR frames to reconstruct the geometric properties of 2D Gaussian splatting and optimizes the LiDAR-camera extrinsic parameters while updating the colors of the 2D Gaussian splats.
• Figure 2: Workflow of the method. The right panel illustrates how the LiDAR frames were used to supervise the geometric properties of the 2DGS. As shown in the left panel, we freeze the geometric properties and update only the color properties during the calibration process. In addition to photometric loss, we also employ two interframe losses: triangulation loss and reprojection loss.
• Figure 3: Example of the rendered depth error. A lighter color indicates a larger value. The depth error map emphasizes the edge components.
• Figure 4: Ground truth image with LiDAR points projected using the resulting extrinsic parameters. An error of several degrees can cause significant displacement, whereas even an error of 20 cm does not result in noticeable displacement. Among the compared methods, the proposed method demonstrated robust results.
• Figure 5: Rendered RGB and depth maps using proposed method and INF. The rendering quality of 2DGS (Ours) was much higher than that of MLP (INF). Because volume rendering samples points within a certain range along a ray, points in distant areas are not sampled, resulting in those areas being unseen in the INF rendered image.