UniCal: Unified Neural Sensor Calibration

TL;DR

This work proposes UniCal, a unified framework for effortlessly calibrating SDVs equipped with multiple LiDARs and cameras that outperforms or matches the accuracy of existing calibration approaches while being more efficient, demonstrating the value of UniCal for scalable calibration.

Abstract

Self-driving vehicles (SDVs) require accurate calibration of LiDARs and cameras to fuse sensor data accurately for autonomy. Traditional calibration methods typically leverage fiducials captured in a controlled and structured scene and compute correspondences to optimize over. These approaches are costly and require substantial infrastructure and operations, making it challenging to scale for vehicle fleets. In this work, we propose UniCal, a unified framework for effortlessly calibrating SDVs equipped with multiple LiDARs and cameras. Our approach is built upon a differentiable scene representation capable of rendering multi-view geometrically and photometrically consistent sensor observations. We jointly learn the sensor calibration and the underlying scene representation through differentiable volume rendering, utilizing outdoor sensor data without the need for specific calibration fiducials. This "drive-and-calibrate" approach significantly reduces costs and operational overhead compared to existing calibration systems, enabling efficient calibration for large SDV fleets at scale. To ensure geometric consistency across observations from different sensors, we introduce a novel surface alignment loss that combines feature-based registration with neural rendering. Comprehensive evaluations on multiple datasets demonstrate that UniCal outperforms or matches the accuracy of existing calibration approaches while being more efficient, demonstrating the value of UniCal for scalable calibration.

Figures (17)

• Figure 1: Our method takes collected data and automatically calibrates the sensor extrinsics. Top: LiDAR-Camera and LiDAR-LiDAR alignment on collected data with uncalibrated extrinsics. Bottom: Sensor alignment with our optimized calibration.
• Figure 2: Overview of our method. We jointly optimize the multi-sensor extrinsics and underlying scene representation within a differentiable framework to minimize the photometric and geometric consistency losses on collected outdoor data retrospectively.
• Figure 3: Overview of surface alignment distance. Ray-casting corresponding pixels $\mathbf{u}_1$ and $\mathbf{u}_2$ into the implicit surface yields 3D points $\mathbf{p}_1$ and $\mathbf{p}_2$. The surface alignment distance quantifies the image-space discrepancy between $\mathbf{p}_1$ and $\mathbf{p}_2$, and minimizing it ensures geometric consistency across sensors or perspectives.
• Figure 4: Visualization of LiDAR-Camera alignment on the checkerboard data. LiDAR points are colored with intensity value.
• Figure 5: Visualization of LiDAR-LiDAR alignment on the flat ground and curb, with each LiDAR represented by a different color.