Label-Efficient Semantic Segmentation of LiDAR Point Clouds in Adverse Weather Conditions

TL;DR

This paper tackles the label scarcity challenge in semantic segmentation of LiDAR point clouds under adverse weather. It introduces a three-stage framework that blends few-shot semantic segmentation, semi-supervised learning with pseudo-labels, and incorporation of good-weather data to robustly segment noise like snow, fog, and spray. Stage Zero initializes from a small set of adversarial-weather labels, Stage One expands via pseudo-labels and SSL, and Stage Two fuses FSS, SSL, and SL with data mixing to perform well across weather conditions. Experiments on multiple real and synthetic datasets show the method achieves competitive performance with fully supervised approaches while using orders of magnitude fewer labels, highlighting practical potential for safer autonomous operation in challenging weather.

Abstract

Adverse weather conditions can severely affect the performance of LiDAR sensors by introducing unwanted noise in the measurements. Therefore, differentiating between noise and valid points is crucial for the reliable use of these sensors. Current approaches for detecting adverse weather points require large amounts of labeled data, which can be difficult and expensive to obtain. This paper proposes a label-efficient approach to segment LiDAR point clouds in adverse weather. We develop a framework that uses few-shot semantic segmentation to learn to segment adverse weather points from only a few labeled examples. Then, we use a semi-supervised learning approach to generate pseudo-labels for unlabelled point clouds, significantly increasing the amount of training data without requiring any additional labeling. We also integrate good weather data in our training pipeline, allowing for high performance in both good and adverse weather conditions. Results on real and synthetic datasets show that our method performs well in detecting snow, fog, and spray. Furthermore, we achieve competitive performance against fully supervised methods while using only a fraction of labeled data.

Figures (4)

• Figure 1: Our proposed method employs three data sources to learn to segment LiDAR scans in adverse weather. First, we use labeled point clouds recorded in adverse weather. As this type of data is expensive and time-consuming to obtain, we use only a few of them. Second, we use unlabeled data in adverse weather, which is cheap and easy to obtain. Third, we use labeled data in good weather conditions, which can be found in many publicly available datasets. We train using a combination of few-shot semantic segmentation (FSS), semi-supervised learning (SSL), and supervised learning (SL).
• Figure 2: Overview of our method. Stage One: we aim to train a model $\phi_{S_1}$ that performs well in adverse weather scenes. We use few-shot semantic segmentation (FSS) to learn from the labeled dataset $\mathcal{D}_t^\ell$ (adverse weather). We iteratively select the best-performing model $\phi_{S_1}^*$ by validating the performance of $\phi_{S_1}$ on the pseudo-validation set $\mathcal{D}_{S_1}^v$. To obtain additional training data without increasing the labeling effort, we generate pseudo-labels for the unlabelled target dataset $\mathcal{D}_t^u$ using $\phi_{S_1}^*$ and train with them in a semi-supervised learning (SSL) fashion. Stage Two: we aim to train a new model $\phi_{S_2}$ that performs well in both good and adverse weather. We use FSS training like in the previous stage. Additionally, we combine pseudo-labels generated from the resulting model $\phi_{S_1}^*$ of Stage One with labeled data sampled from $\mathcal{D}_s$ (good weather) using polar data mixing and train with them using a combination of supervised-learning (SL) and SSL. We select the best-performing model $\phi_{S_2}^*$ by validating $\phi_{S_2}$ on the pseudo-validation set $\mathcal{D}_{S_2}^v$.
• Figure 3: Overview of the pseudo-validation sets generation. Top: to generate the pseudo-validation set for Stage One $\mathcal{D}_{S_1}^v$ we use scans from the $K$ labeled data in $\mathcal{D}_t^\ell$, augment them, and then use polar mixing to generate new point clouds. Bottom: for the pseudo-validation set for Stage Two $\mathcal{D}_{S_2}^v$, we additionally use data in good weather sampled from $\mathcal{D}_s$.
• Figure 4: Qualitative comparison results on different datasets. The results refer to FSSAD (Baseline) and FSSAD trained with our method (Ours). All results are obtained with $K=1$. For each method, we show on the left a binary-coded visualization of $\color{bin-correct}{\bullet}~$correctly and $\color{bin-wrong}{\bullet}~$incorrectly segmented points and on the right the class predictions. The top row refers to the SemanticSpray dataset, with colormap: $\color{SS-background}{\bullet}~$background$\color{SS-car}{\bullet}~$car$\color{SS-spray}{\bullet}~$spray. The bottom row refers to the SemanticKITTI-fog dataset with colormap: $\color{car}{\bullet}~$car$\color{road}{\bullet}~$road$\color{sidewalk}{\bullet}~$sidewalk$\color{other-ground}{\bullet}~$other-ground$\color{building}{\bullet}~$building$\color{fence}{\bullet}~$fence$\color{vegetation}{\bullet}~$vegetation$\color{trunk}{\bullet}~$trunk$\color{terrain}{\bullet}~$terrain$\color{fog}{\bullet}~$fog.