DST-Calib: A Dual-Path, Self-Supervised, Target-Free LiDAR-Camera Extrinsic Calibration Network

TL;DR

DST-Calib tackles LiDAR-Camera extrinsic calibration without targets, enabling online self-supervised operation. It identifies generalization gaps caused by single-sided data augmentation and remedies them with double-sided augmentation that generates multi-view camera perspectives from depth-based projections using Depth Anchor Refinement. The method adopts a single-branch, difference-map architecture for cross-modal correlation and supports fully supervised and self-supervised online calibration, with multi-frame optimization to refine extrinsic parameters. Extensive experiments across seven public/private datasets, including a newly released LCScenes, show state-of-the-art generalization and robustness, including zero-shot cross-domain performance and online applicability.

Abstract

LiDAR-camera extrinsic calibration is essential for multi-modal data fusion in robotic perception systems. However, existing approaches typically rely on handcrafted calibration targets (e.g., checkerboards) or specific, static scene types, limiting their adaptability and deployment in real-world autonomous and robotic applications. This article presents the first self-supervised LiDAR-camera extrinsic calibration network that operates in an online fashion and eliminates the need for specific calibration targets. We first identify a significant generalization degradation problem in prior methods, caused by the conventional single-sided data augmentation strategy. To overcome this limitation, we propose a novel double-sided data augmentation technique that generates multi-perspective camera views using estimated depth maps, thereby enhancing robustness and diversity during training. Built upon this augmentation strategy, we design a dual-path, self-supervised calibration framework that reduces the dependence on high-precision ground truth labels and supports fully adaptive online calibration. Furthermore, to improve cross-modal feature association, we replace the traditional dual-branch feature extraction design with a difference map construction process that explicitly correlates LiDAR and camera features. This not only enhances calibration accuracy but also reduces model complexity. Extensive experiments conducted on five public benchmark datasets, as well as our own recorded dataset, demonstrate that the proposed method significantly outperforms existing approaches in terms of generalizability.

Figures (14)

• Figure 1: Our proposed DST-Calib estimates the extrinsic transformation with six degrees of freedom (6-DoF) between LiDAR scans and camera images in the wild. It can be readily employed for robotic tasks such as object detection, odometry, and localization.
• Figure 2: Mapping relationship of different data augmentation methods in the training process of LCEC network: (a) the classical single-sided data augmentation with many-to-one mapping; (b) our proposed double-sided approach with many-to-many mapping.
• Figure 3: The comparison of the DNN-based LiDAR-camera extrinsic calibration pipelines: (a) Classical double-branch architecture with single-sided data augmentation; (b) Our proposed single-branch architecture with double-sided data augmentation.
• Figure 4: Depth refinement via LiDAR-camera depth anchor reconstruction.
• Figure 5: Qualitative comparisons between the initial estimated depth cloud and those after the correction of our DAR on various datasets: (a)-(d) RGB images, initial estimated depth cloud, corrected depth cloud, and corresponding point cloud scanned by LiDARs.