Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather

TL;DR

LiDAR semantic segmentation suffers when adverse weather degrades signal quality. The authors perform a data-centric analysis and identify two key distortion types—geometric perturbation and point drop—as primary drivers of performance loss, then introduce Selective Jittering and Learnable Point Drop to simulate these effects during training. The approach yields state-of-the-art robustness on SemanticSTF, achieving an $+8.1$ $mIoU$ improvement over the baseline and generalizing across backbones and weather conditions without explicit weather modeling. This work offers a scalable, weather-agnostic augmentation strategy with practical impact for safe autonomous perception, and the authors plan to release code.

Abstract

Existing LiDAR semantic segmentation methods often struggle with performance declines in adverse weather conditions. Previous work has addressed this issue by simulating adverse weather or employing universal data augmentation during training. However, these methods lack a detailed analysis and understanding of how adverse weather negatively affects LiDAR semantic segmentation performance. Motivated by this issue, we identified key factors of adverse weather and conducted a toy experiment to pinpoint the main causes of performance degradation: (1) Geometric perturbation due to refraction caused by fog or droplets in the air and (2) Point drop due to energy absorption and occlusions. Based on these findings, we propose new strategic data augmentation techniques. First, we introduced a Selective Jittering (SJ) that jitters points in the random range of depth (or angle) to mimic geometric perturbation. Additionally, we developed a Learnable Point Drop (LPD) to learn vulnerable erase patterns with a Deep Q-Learning Network to approximate the point drop phenomenon from adverse weather conditions. Without precise weather simulation, these techniques strengthen the LiDAR semantic segmentation model by exposing it to vulnerable conditions identified by our data-centric analysis. Experimental results confirmed the suitability of the proposed data augmentation methods for enhancing robustness against adverse weather conditions. Our method achieves a notable 39.5 mIoU on the SemanticKITTI-to-SemanticSTF benchmark, improving the baseline by 8.1\%p and establishing a new state-of-the-art. Our code will be released at \url{https://github.com/engineerJPark/LiDARWeather}.

Figures (5)

• Figure 1: Visualization for the results of toy experiment. In all figures, green (or red) points indicate correct (or incorrect) predictions. Point drop, occlusion, and geometric perturbation are the main distortions that lead to performance degradation. Also, since the misprediction areas of point drop and occlusion largely overlap, it's reasonable to consider them as the same type of distortion.
• Figure 2: Description of Learnable Point Drop (LPD). (1) The LPD utilizes the loss $L_{aug}$ and entropy $H_{aug}$ derived from LiDAR segmentation model as its current state. (2) Upon receiving a point input, LPD performs a point drop, then recalculates the loss $L_{LPD}$ and entropy $H_{LPD}$. (3) The difference between these new values and the previous ones is used to compute the reward.
• Figure 3: The overall training process. (1) Initially, SJ is applied. (2) Following this, we calculate the loss, which leads to the update of the LPD state. (3) This process is followed by utilizing LPD to generate dropped points. (4) Subsequent to recalculating the loss, the reward is accordingly updated.
• Figure 4: Qualitative results of our method from validation set of SemanticSTF. All models are trained in train set of SemanticKITTI. Green (or red) points indicate correct (or incorrect) predictions, showing our predictions are more accurate than PointDR in all weather, especially on road and sidewalk.
• Figure 5: Qualitative results of LPD from validation set of SemanticKITTI. It can be observed that the point drop facilitated by the LPD occurs not just locally but extends across all points. While LPD effectively mimics the occlusions and sparse point distributions seen in real adverse weather, it does not replicate the effect of wet ground surfaces.