Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection

TL;DR

This work tackles the challenge of LiDAR-based 3D object detection under adverse weather by addressing data scarcity and weather-induced domain gaps. It introduces DRET, a two-stage rain simulation that fuses dynamic splash modeling with rainy-environment theory to produce realistic rain-impacted point clouds, and SRKD, a Sunny-to-Rainy Knowledge Distillation framework comprising AWID, PRD, and NAPC to transfer knowledge from sunny to rainy conditions while correcting rain noise. Across Waymo Open Dataset experiments and multiple detectors, the authors demonstrate consistent improvements in rainy robustness and even modest gains in sunny conditions, validating the framework's versatility and practicality. The combination of realistic rain data augmentation and cross-weather distillation offers a scalable path toward robust 3D perception for autonomous driving in diverse weather scenarios.

Abstract

LiDAR-based 3D object detection models have traditionally struggled under rainy conditions due to the degraded and noisy scanning signals. Previous research has attempted to address this by simulating the noise from rain to improve the robustness of detection models. However, significant disparities exist between simulated and actual rain-impacted data points. In this work, we propose a novel rain simulation method, termed DRET, that unifies Dynamics and Rainy Environment Theory to provide a cost-effective means of expanding the available realistic rain data for 3D detection training. Furthermore, we present a Sunny-to-Rainy Knowledge Distillation (SRKD) approach to enhance 3D detection under rainy conditions. Extensive experiments on the WaymoOpenDataset large-scale dataset show that, when combined with the state-of-the-art DSVT model and other classical 3D detectors, our proposed framework demonstrates significant detection accuracy improvements, without losing efficiency. Remarkably, our framework also improves detection capabilities under sunny conditions, therefore offering a robust solution for 3D detection regardless of whether the weather is rainy or sunny

Figures (7)

• Figure 1: Visualization of rain noise points in groundtruth and different rain simulations.
• Figure 2: Comparison of the performance under both sunny and rainy weather conditions on the WOD.
• Figure 3: The overview of our method, including (a) DRET for rain simulation and (b) SRKD framework for 3D object detection. DRET involves a two-stage process. The first stage simulates the dynamic splashes to generate rain particles, and the second stage matches rain particles with the point cloud and processes the scene based on the rainy environment theory. SRKD enables sunny-to-rainy knowledge distillation with the help of (c) AWID and PRD, which avoids the problems associated with the weather domain gap. PRD is a classic response distillation with some adjustments, which will be introduced in the SRKD method. Additionally, the (d) NAPC module is designed to mitigate the influence of rain-induced noise.
• Figure 4: The average intensity gap. (b) The average points gap in each distance interval.
• Figure 5: Examples of rainy condition effects on 3D object detection. The blue and red boxes represent groundtruths and predictions.