Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles

TL;DR

This work tackles the challenge of sensor misalignment degrading long-range perception in autonomous vehicles. It introduces a generic multi-task learning framework that jointly estimates angular miscalibration $\Delta r=[\theta^{\text{roll}},\theta^{\text{pitch}},\theta^{\text{yaw}}]$, predicts calibrated uncertainty, and self-corrects LiDAR-camera extrinsics to improve long-range 3D detection. The method operates in three phases: Phase 1 pre-processing to create a depth image, Phase 2 inference unifies misalignment estimation with 3D detections, and Phase 3 post-processing fuses misalignment estimates over time and applies self-correction. Evaluations on a fault-injected, long-range internal dataset (up to $500\,\text{m}$) show robust misalignment detection, improved BEV detection when applying predicted corrections, and favorable comparisons to baselines, highlighting practical benefits for robust fusion in real-world AV systems.

Abstract

Advances in machine learning algorithms for sensor fusion have significantly improved the detection and prediction of other road users, thereby enhancing safety. However, even a small angular displacement in the sensor's placement can cause significant degradation in output, especially at long range. In this paper, we demonstrate a simple yet generic and efficient multi-task learning approach that not only detects misalignment between different sensor modalities but is also robust against them for long-range perception. Along with the amount of misalignment, our method also predicts calibrated uncertainty, which can be useful for filtering and fusing predicted misalignment values over time. In addition, we show that the predicted misalignment parameters can be used for self-correcting input sensor data, further improving the perception performance under sensor misalignment.

Figures (4)

• Figure 1: Demonstration of sensor misalignment correction using the proposed approach. The top image shows an input frame with visible misalignment when projecting the LiDAR data (green points) onto the camera image. In the bottom image, after applying the extrinsic parameter correction predicted by our model, the LiDAR points are well-aligned with the camera data, leading to improved 3D object detection accuracy.
• Figure 2: Proposed system architecture for robust long-range perception against sensor misalignment. In the figure, rectangles indicate the system inputs, while circles denote the outputs.
• Figure 3: Our proposed multi-task network architecture and data augmentation approach. The network takes in RGB image and LiDAR points projected onto a virtual image. During the training, projected LiDAR points are perturbed in a controlled fashion, and the network is tasked to predict the parameters of that perturbation along with detecting and classifying 3D objects in the scene. This synthetic data augmentation and training technique ensures the model is robust to small perturbations in real-world scenarios.
• Figure 4: Mean and variance observed in misalignment estimation error at different levels of perturbations for each rotational dimension.