Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention

TL;DR

This work tackles semantic segmentation under sequential, unlabeled adverse weather conditions by proposing a continual unsupervised domain adaptation framework. It introduces three mechanisms—adaptive knowledge acquisition, pseudo-label blending, and weather composition replay—to learn new weather domains while preserving previously acquired knowledge. The approach uses model- and feature-level signals, a previous-teacher ensemble for pseudo-labels, and frequency-domain weather vector replay to mitigate forgetting. Empirical results on Cityscapes to ACDC across four weather targets show superior average mIoU (65.7%) and reduced forgetting (3.6%) compared with state-of-the-art baselines, demonstrating robust cross-domain adaptation with practical impact for autonomous driving under diverse conditions.

Abstract

Semantic segmentation's performance is often compromised when applied to unlabeled adverse weather conditions. Unsupervised domain adaptation is a potential approach to enhancing the model's adaptability and robustness to adverse weather. However, existing methods encounter difficulties when sequentially adapting the model to multiple unlabeled adverse weather conditions. They struggle to acquire new knowledge while also retaining previously learned knowledge.To address these problems, we propose a semantic segmentation method for multiple adverse weather conditions that incorporates adaptive knowledge acquisition, pseudolabel blending, and weather composition replay. Our adaptive knowledge acquisition enables the model to avoid learning from extreme images that could potentially cause the model to forget. In our approach of blending pseudo-labels, we not only utilize the current model but also integrate the previously learned model into the ongoing learning process. This collaboration between the current teacher and the previous model enhances the robustness of the pseudo-labels for the current target. Our weather composition replay mechanism allows the model to continuously refine its previously learned weather information while simultaneously learning from the new target domain. Our method consistently outperforms the stateof-the-art methods, and obtains the best performance with averaged mIoU (%) of 65.7 and the lowest forgetting (%) of 3.6 against 60.1 and 11.3, on the ACDC datasets for a four-target continual multi-target domain adaptation.

Figures (4)

• Figure 1: The illustration of our method, where the model adapts to each target domain sequentially. MIC hoyer2023mic fails to retain previously learned knowledge, as its performance on the first target gradually deteriorates, e.g., the sky and the side walk in Target 1 are disappearing after the method learns Targets 2 and 3. Our method retain previously learned knowledge while adapting to new targets.
• Figure 2: Our architecture for adapting a model to $n$ adverse weather conditions in $n$ steps in a sequential manner. The architecture consists of several key components: (1) Adaptive knowledge acquisition, where the model is guided to avoid learning the areas that could lead to a forgetting problem. (2) Pseudo-label blending, where the previous teacher is involved for enhancing the pseudo-label. (3) Weather composition replay, where the weather vectors from previous steps are composed into the current target image for revising on previously learned knowledge.
• Figure 3: Examples of composing night and fog weather vectors into snow and rain images, respectively.
• Figure : Image MIC hoyer2023mic Ours Ground Truth