Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning

Marcell Kegl, Andras Palffy, Csaba Benedek, Dariu M. Gavrila

Abstract

In this paper, we address extrinsic calibration for camera, lidar, and 4D radar sensors. Accurate extrinsic calibration of radar remains a challenge due to the sparsity of its data. We propose CLRNet, a novel, multi-modal end-to-end deep learning (DL) calibration network capable of addressing joint camera-lidar-radar calibration, or pairwise calibration between any two of these sensors. We incorporate equirectangular projection, camera-based depth image prediction, additional radar channels, and leverage lidar with a shared feature space and loop closure loss. In extensive experiments using the View-of-Delft and Dual-Radar datasets, we demonstrate superior calibration accuracy compared to existing state-of-the-art methods, reducing both median translational and rotational calibration errors by at least 50%. Finally, we examine the domain transfer capabilities of the proposed network and baselines, when evaluating across datasets. The code will be made publicly available upon acceptance at: https://github.com/tudelft-iv.

CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning

Abstract

In this paper, we address extrinsic calibration for camera, lidar, and 4D radar sensors. Accurate extrinsic calibration of radar remains a challenge due to the sparsity of its data. We propose CLRNet, a novel, multi-modal end-to-end deep learning (DL) calibration network capable of addressing joint camera-lidar-radar calibration, or pairwise calibration between any two of these sensors. We incorporate equirectangular projection, camera-based depth image prediction, additional radar channels, and leverage lidar with a shared feature space and loop closure loss. In extensive experiments using the View-of-Delft and Dual-Radar datasets, we demonstrate superior calibration accuracy compared to existing state-of-the-art methods, reducing both median translational and rotational calibration errors by at least 50%. Finally, we examine the domain transfer capabilities of the proposed network and baselines, when evaluating across datasets. The code will be made publicly available upon acceptance at: https://github.com/tudelft-iv.
Paper Structure (30 sections, 5 equations, 3 figures, 4 tables)

This paper contains 30 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Targetless Extrinsic Calibration
  4. End-to-End Deep Learning for Lidar–Camera Calibration
  5. Targetless Calibration Involving Radar
  6. Targetless Calibration Involving 3D Radar
  7. Targetless Calibration Involving 4D Radar
  8. Methodology
  9. Problem Formulation and Data Preprocessing
  10. Equirectangular Projection
  11. Network Architecture
  12. Feature Extraction
  13. Feature Matching
  14. Parameter Regression from a Shared Feature Space
  15. Loss Function
  16. ...and 15 more sections

Figures (3)

  • Figure 1: CLRNet network architecture overview. On the lidar and radar branch, the input point clouds are projected using equirectangular projection before (a) feature extraction. After the pairwise (b) feature matching layers, we (c) concatenate the output to have shared information for the (d) parameter regression layers, which are used to predict the final extrinsic parameters for each three sensor pairs.
  • Figure 2: Qualitative results of lidar point projection to the camera image with two reference methods LCCNetHayoun_clr_selfsup and the proposed CLRNet model. CLRNet is superior to Hayoun_clr_selfsup, also using three sensors; while our approach similarly performs to LCCNet, which is a pairwise lidar-camera calibration baseline method.
  • Figure 3: Qualitative results of radar point projection to the camera image in three test scenes by using the calibration parameters of the (b) ground truth, (c)-(e) the three reference methods Scholler_radar_DLHayoun_clr_selfsup4DRC-OC and (f) the proposed CLRNet model.