Generalizable Autonomous Driving System across Diverse Adverse Weather Conditions

TL;DR

AdvImmu tackles the challenge of semantic segmentation for autonomous driving under diverse adverse weather without relying on reference images. It combines Locally Sequential Mechanism (LSM), Globally Shuffled Mechanism (GSM), and Unfolded Regularizers (URs) to leverage temporal consistency while preventing overfitting to sequence order; regularizers are unfolded into layers for stable optimization. To train without frame-wise manual annotations, it integrates Segment Anything Model (SAM) with the SBICAC clustering algorithm to produce category-aware pseudo-labels. Extensive experiments across six datasets demonstrate that AdvImmu achieves substantial improvements over SOTA in mIoU and generalizes well to mixed adverse conditions, with a justifiable computational cost driven by unfolding. The work highlights a practical path toward robust, reference-free AD perception and points to future extensions to other tasks and collaborative learning frameworks.

Abstract

Various adverse weather conditions pose a significant challenge to autonomous driving (AD) street scene semantic understanding (segmentation). A common strategy is to minimize the disparity between images captured in clear and adverse weather conditions. However, this technique typically relies on utilizing clear image as a reference, which is challenging to obtain in practice. Furthermore, this method typically targets a single adverse condition, and thus perform poorly when confronting a mixture of multiple adverse weather conditions. To address these issues, we introduce a reference-free and Adverse weather-Immune scheme (called AdvImmu) that leverages the invariance of weather conditions over short periods (seconds). Specifically, AdvImmu includes three components: Locally Sequential Mechanism (LSM), Globally Shuffled Mechanism (GSM), and Unfolded Regularizers (URs). LSM leverages temporal correlations between adjacent frames to enhance model performance. GSM is proposed to shuffle LSM segments to prevent overfitting of temporal patterns. URs are the deep unfolding implementation of two proposed regularizers to penalize the model complexity to enhance across-weather generalization. In addition, to overcome the over-reliance on consecutive frame-wise annotations in the training of AdvImmu (typically unavailable in AD scenarios), we incorporate a foundation model named Segment Anything Model (SAM) to assist to annotate frames, and additionally propose a cluster algorithm (denoted as SBICAC) to surmount SAM's category-agnostic issue to generate pseudo-labels. Extensive experiments demonstrate that the proposed AdvImmu outperforms existing state-of-the-art methods by 88.56% in mean Intersection over Union (mIoU).

Figures (15)

• Figure 1: Major challenges in the realm of domain adaption li2023vblc. (a) Requirement of reference images. Reference images, highlighted in the red box located at the bottom right corner, depict clear-weather-condition scenes which are almost same with those under adverse weather conditions (e.g., nighttime or fog). Generally, it is difficult to collect such reference images in practice. (b) Poor performance under the hybrid of multiple adverse conditions. In general, domain adaption models always work well in adapted target domains and face significant performance decline in unadapted domains, which results in poor performance when dealing with the case of hybrid of multiple adverse weather conditions.
• Figure 2: Overview of the proposed AdvImmu. In the context of LSM, Instant Unit (InsU), Integral Unit (IntU), and Derivative Unit (DU) capture instantaneous information, stable and shared background information, and dynamic changes from the input, respectively. These three units work together to enhance model performance by leveraging temporal correlation of intra-weather condition. Notably, $T$ denotes the current frame number in the input sequence, while $t$ (i.e., LSM depth) refers to the count of consecutive frames prior to frame $T$. GSM shuffles and groups the LSM segments to avoid the overfitting to specific temporal patterns, whereas GSM-Free does not. GSM is proposed to enable the model to learn common patterns across multiple adverse weather conditions. URs include two unfolded regularizers and are proposed to penalize the model complexity to avoid overfitting to specific patterns as well, therefore, enhancing the model performance across multiple adverse weather conditions.
• Figure 3: Illustration of the category-agnostic issue in SAM-generated mask. For example, trees are assigned to different semantic IDs (colors) in SAM-generated mask.
• Figure 4: Illustration of LF.
• Figure 5: Illustration of AdvImmu architecture, which contains LSaGS and URs. (4), (5), (6) represent equation number.