Targetless LiDAR-Camera Calibration with Neural Gaussian Splatting

TL;DR

TLC-Calib addresses targetless LiDAR–camera calibration by jointly optimizing extrinsics and a neural Gaussian scene representation. It introduces anchor Gaussians to fix global structure and auxiliary Gaussians as learnable buffers to model local details, all within a differentiable pipeline that propagates photometric and geometric cues. Across KITTI-360, Waymo, and Fast-LIVO2, TLC-Calib achieves superior calibration accuracy and novel-view synthesis quality, while demonstrating strong cross-scene generalization without reliance on markers or region cropping. This work enables robust, scalable, and deployment-friendly multi-sensor calibration for real-world autonomous systems.

Abstract

Accurate LiDAR-camera calibration is crucial for multi-sensor systems. However, traditional methods often rely on physical targets, which are impractical for real-world deployment. Moreover, even carefully calibrated extrinsics can degrade over time due to sensor drift or external disturbances, necessitating periodic recalibration. To address these challenges, we present a Targetless LiDAR-Camera Calibration (TLC-Calib) that jointly optimizes sensor poses with a neural Gaussian-based scene representation. Reliable LiDAR points are frozen as anchor Gaussians to preserve global structure, while auxiliary Gaussians prevent local overfitting under noisy initialization. Our fully differentiable pipeline with photometric and geometric regularization achieves robust and generalizable calibration, consistently outperforming existing targetless methods on KITTI-360, Waymo, and FAST-LIVO2, and surpassing even the provided calibrations in rendering quality.

Figures (6)

• Figure 1: Comparison of optimization landscapes. (a) Our naı̈ve attempt (3DGS kerbl20233dgs + camera rig optimization) appears to converge to a global minimum within its own loss landscape, but the landscape is ambiguous and misaligned with the true geometry, leading to convergence to an incorrect pose far from the dataset calibration. (b) Our method produces a smoother and more geometrically consistent landscape by leveraging anchor and auxiliary Gaussians, enabling stable convergence toward the correct pose closer to the dataset calibration and demonstrating more reliable optimization behavior.
• Figure 2: Overview of the TLC-Calib pipeline. After aggregating LiDAR scans into a globally aligned point cloud, anchor Gaussians serve as fixed geometric references (their positions are not optimized), while auxiliary Gaussians adapt to local geometry and guide extrinsic optimization through photometric loss. Unlike anchor Gaussians, auxiliary Gaussians serve as learnable buffers around anchors, helping the optimization avoid local minima. Additionally, a rig-based optimization strategy jointly refines all cameras in relation to the scene, ensuring consistent and stable calibration across views.
• Figure 3: Box plots of calibration accuracy: (a) rotation errors and (b) translation errors on KITTI-360 scenes over 10 trials. Our method consistently yields lower errors than the baselines for both front and side cameras, with particularly large improvements on side views. For clarity, the first y-axis interval is enlarged by a factor of two.
• Figure 4: Qualitative comparison on Waymo. Key improvements are highlighted with yellow boxes, and cropped patches show zoomed-in regions for clarity. PSNR values of the rendered outputs are shown in the top-right corner of each image.
• Figure 5: Qualitative evaluation of LiDAR-camera alignment accuracy with KITTI-360 dataset. LiDAR points are projected onto images using the calibration results estimated by each baseline method. Point colors indicate 3D distances from the LiDAR, ranging from red (near) to blue (far).