PseudoCal: Towards Initialisation-Free Deep Learning-Based Camera-LiDAR Self-Calibration

TL;DR

This work tackles camera-LiDAR extrinsic calibration by removing the need for an initial pose estimate. It introduces PseudoCal, a cascaded architecture that first computes a coarse alignment with a PseudoPillars module using pseudo-LiDAR derived from monocular depth, then refines the result with two UniCal networks trained on decreasing decalibration ranges. The approach integrates a GPLN-based depth model, a Pillars encoder, and a MobileViT backbone, aided by Canny-based depth edge filtering, and trained with artificial decalibration and augmentation, achieving initialization-free calibration on KITTI, including extreme yaw decalibrations up to $\pm 180^\circ$. This has practical impact for on-the-fly recalibration and faster sensor integration in autonomous systems, enabling robust fusion without manual targets or good starting guesses. Overall, PseudoCal advances deep learning-based self-calibration by leveraging pseudo-LiDAR in 3D space and a lightweight, cascade-based refinement, demonstrating competitive performance and resilience to challenging decalibrations.

Abstract

Camera-LiDAR extrinsic calibration is a critical task for multi-sensor fusion in autonomous systems, such as self-driving vehicles and mobile robots. Traditional techniques often require manual intervention or specific environments, making them labour-intensive and error-prone. Existing deep learning-based self-calibration methods focus on small realignments and still rely on initial estimates, limiting their practicality. In this paper, we present PseudoCal, a novel self-calibration method that overcomes these limitations by leveraging the pseudo-LiDAR concept and working directly in the 3D space instead of limiting itself to the camera field of view. In typical autonomous vehicle and robotics contexts and conventions, PseudoCal is able to perform one-shot calibration quasi-independently of initial parameter estimates, addressing extreme cases that remain unsolved by existing approaches.

Figures (6)

• Figure 1: Illustration of the PseudoPillars module, key component of the PseudoCal method allowing for calibration estimation quasi-independently from initial values.
• Figure 2: Illustration of the method on driving scenes from KITTI. Points are a depth-coloured LiDAR projection. On top are decalibrated samples in two extreme scenarios where existing deep learning-based methods fail. Bottom visuals represent PseudoCal's correction. (a) shows no LiDAR-camera overlap due to severe pitch axis decalibration, and (b) presents a severe yaw axis decalibration which results in projected points corresponding to objects outside of the camera field of view.
• Figure 3: Overview of the PseudoCal architecture and its three main components: PseudoPillars, UniCal-M, and UniCal-S. PseudoPillars provides an initial estimation to the cascaded UniCal modules for refinement. The 6D transformation (calibration parameters) is noted T.
• Figure 4: Kitti scene without (left) and with (right) edge removal.
• Figure 5: Visualization of the Pillars representation of a KITTI scene for the LiDAR (top) and the camera (bottom). (a) is calibrated, (b) is decalibrated by $90^\circ$ on yaw for the illustration, (c) has the same decalibration and a $180^\circ$ yaw augmentation.