Automatic Target-Less Camera-LiDAR Calibration From Motion and Deep Point Correspondences

TL;DR

This letter proposes MDPCalib which is a novel method for camera-LiDAR calibration that requires neither human supervision nor any specific target objects, and utilizes sensor motion estimates from visual and LiDAR odometry as well as deep learning-based 2D-pixel-to-3D-point correspondences that are obtained without in-domain retraining.

Abstract

Sensor setups of robotic platforms commonly include both camera and LiDAR as they provide complementary information. However, fusing these two modalities typically requires a highly accurate calibration between them. In this paper, we propose MDPCalib which is a novel method for camera-LiDAR calibration that requires neither human supervision nor any specific target objects. Instead, we utilize sensor motion estimates from visual and LiDAR odometry as well as deep learning-based 2D-pixel-to-3D-point correspondences that are obtained without in-domain retraining. We represent camera-LiDAR calibration as an optimization problem and minimize the costs induced by constraints from sensor motion and point correspondences. In extensive experiments, we demonstrate that our approach yields highly accurate extrinsic calibration parameters and is robust to random initialization. Additionally, our approach generalizes to a wide range of sensor setups, which we demonstrate by employing it on various robotic platforms including a self-driving perception car, a quadruped robot, and a UAV. To make our calibration method publicly accessible, we release the code on our project website at http://calibration.cs.uni-freiburg.de.

Figures (10)

• Figure 1: Our proposed method, called MDPCalib, for camera-LiDAR calibration comprises two steps: We first initialize the extrinsic parameters by aligning the motion of both sensors. Afterward, we refine the calibration results by leveraging deep learning-based 2D-to-3D point correspondences.
• Figure 2: Our proposed method for camera-LiDAR calibration processes two input streams of RGB images and 3D point clouds. The first step comprises a coarse registration based on sensor motion estimated with visual and LiDAR odometry. These motion estimates yield time-synchronized matches serving as constraints in an optimization problem. Given the obtained initial calibration parameters, a neural network is used to find 2D pixel to 3D point correspondences that result in additional constraints. The second step consists of joint optimization with respect to both sensor motion and point correspondences yielding the overall extrinsic calibration parameters.
• Figure 3: In the fine-tuning stage, we employ CMRNext cattaneo2024cmrnext to find 2D pixel to 3D point correspondences. First, a LiDAR point is projected onto the image space using the coarse calibration parameters. Second, CMRNext predicts a 2D offset to correct the projection. Finally, during optimization, the calibration parameters are adjusted to match the corrected projection.
• Figure 4: We interpolate the poses from LiDAR odometry to the timestamps of the camera poses and further project the point cloud of the nearest neighbor to the same time to yield synchronized image-point cloud pairs. The odometry poses are then used to align the sensor motion, whereas the sensor measurements are fed to CMRNext cattaneo2024cmrnext to obtain point correspondences.
• Figure 5: We calibrate the sensors on three in-house robotic platforms including a self-driving perception car, a quadruped robot, and a UAV.