Robust Single Object Tracking in LiDAR Point Clouds under Adverse Weather Conditions

TL;DR

The paper introduces a weather-robust benchmarking suite for LiDAR-based 3D single object tracking, comprising synthetic datasets KITTI-A and nuScenes-A and a real-world CADC-SOT dataset across rain, fog, and snow. It analyzes three degradation factors—target distance, template shape corruption, and target shape corruption—and demonstrates substantial performance drops of state-of-the-art trackers under adverse weather. To address this, it proposes DRCT, a dual-branch training framework that combines domain randomization with local geometric contrast learning to transfer robustness from a randomized auxiliary domain to the clean primary domain. The results on synthetic and real data show that DRCT improves robustness while preserving normal-weather performance, offering a practical path toward safer, reliable 3DSOT in real-world, weather-affected outdoor environments.

Abstract

3D single object tracking (3DSOT) in LiDAR point clouds is a critical task for outdoor perception, enabling real-time perception of object location, orientation, and motion. Despite the impressive performance of current 3DSOT methods, evaluating them on clean datasets inadequately reflects their comprehensive performance, as the adverse weather conditions in real-world surroundings has not been considered. One of the main obstacles is the lack of adverse weather benchmarks for the evaluation of 3DSOT. To this end, this work proposes a challenging benchmark for LiDAR-based 3DSOT in adverse weather, which comprises two synthetic datasets (KITTI-A and nuScenes-A) and one real-world dataset (CADC-SOT) spanning three weather types: rain, fog, and snow. Based on this benchmark, five representative 3D trackers from different tracking frameworks conducted robustness evaluation, resulting in significant performance degradations. This prompts the question: What are the factors that cause current advanced methods to fail on such adverse weather samples? Consequently, we explore the impacts of adverse weather and answer the above question from three perspectives: 1) target distance; 2) template shape corruption; and 3) target shape corruption. Finally, based on domain randomization and contrastive learning, we designed a dual-branch tracking framework for adverse weather, named DRCT, achieving excellent performance in benchmarks.

Figures (7)

• Figure 1: Visualization of each weather type in our KITTI-A.
• Figure 2: Visualization of each level snowy in our CADC-SOT.
• Figure 3: Analysis of the mean IOU deviation in KITTI-A. The same column represents the same weather and category, and the same row represents the same factor. There are three factors: Target Distance, the horizontal axis represents the target distance; Template Corruption, the horizontal coordinates represent the Hausdorff distance between point clouds of templates in normal and adverse weather for the same tracking sequence; Target Corruption, the horizontal coordinates represent the Hausdorff distance between point clouds in normal and adverse weather for the same target. The vertical axis illustrates the IOU deviation between normal and adverse weather.
• Figure 4: The analysis in CADC-SOT, the horizontal axis represents the target distance, while the vertical axis illustrates the success or precision.
• Figure 5: The overview of the proposed DRCT. DRCT consists of a primary branch (above) and an auxiliary branch (below), with a shared input of the point cloud $P_t$ at time $t$. The primary branch performs feature extraction on $P_t$ to obtain features $F_t$ and down-sampled points $\Tilde{P}_t$, while the auxiliary branch first generates a randomized point cloud $\mathcal{P}_t$ through domain randomization, then extracts its features $\mathcal{F}_t$ and down-sampled points $\Tilde{\mathcal{P}}_t$. The features from both branches are linked through local geometric contrastive learning, with $C_i$ being the common key points of both branches and neighborhood features are aggregated for contrastive learning.