Domain-Incremental Semantic Segmentation for Autonomous Driving under Adverse Driving Conditions

TL;DR

Semantic segmentation for autonomous driving degrades under adverse conditions and faces catastrophic forgetting when learning new domains. The paper introduces Progressive Semantic Segmentation (PSS), an architecture-based domain-incremental framework that grows domain-specific segmentation experts and uses a collection of autoencoders to infer the domain and route inputs to the appropriate expert. PSS demonstrates strong performance, matching or closely approaching joint training while avoiding forgetting across real-world and unseen domains, and extends to hybrid incremental learning and transfer to object detection. The approach offers practical benefits for continual adaptation in autonomous systems with only a small inference overhead and flexible domain modeling; future work could integrate domain adaptation techniques to further leverage past knowledge for new tasks.

Abstract

Semantic segmentation for autonomous driving is an even more challenging task when faced with adverse driving conditions. Standard models trained on data recorded under ideal conditions show a deteriorated performance in unfavorable weather or illumination conditions. Fine-tuning on the new task or condition would lead to overwriting the previously learned information resulting in catastrophic forgetting. Adapting to the new conditions through traditional domain adaption methods improves the performance on the target domain at the expense of the source domain. Addressing these issues, we propose an architecture-based domain-incremental learning approach called Progressive Semantic Segmentation (PSS). PSS is a task-agnostic, dynamically growing collection of domain-specific segmentation models. The task of inferring the domain and subsequently selecting the appropriate module for segmentation is carried out using a collection of convolutional autoencoders. We extensively evaluate our proposed approach using several datasets at varying levels of granularity in the categorization of adverse driving conditions. Furthermore, we demonstrate the generalization of the proposed approach to similar and unseen domains.

Figures (9)

• Figure 1: Progressive Semantic Segmentation (PSS) continually learns to handle adverse conditions. Our proposed approach accommodates to changing weather and illumination conditions by first inferring the domain and subsequently using a domain expert for segmentation.
• Figure 2: Overview of the proposed Progressive Semantic Segmentation (PSS). For each task-increment $T_k$ and the associated data $D_k=(X_k, Y_k)$, we train a task-specific autoencoder $AE_k$ using $X_k$ and a segmentation model $S_k$. During inference, the test image $x$ is reconstructed using autoencoders from all tasks, and the reconstruction losses are computed. The domain is inferred from the autoencoder with the lowest loss and the image is routed to the corresponding segmentation model.
• Figure 3: Proposed autoencoder architecture with four-layer deep encoder and decoder. Domain inference is based on the difference between reconstructed and input image.
• Figure 4: Classification results for domain inference. Left: SHIFT treats each adversarial condition as a separate domain, resulting in a multi-class classification. Right: Real vs synthetic data both representing daytime conditions, adds complexity to classification.
• Figure 5: Examples of images for Cityscapes generated by GANformer ganformer and the corresponding pseudo labels using the CS domain expert. The sample on the left is reasonably accurate, while the right sample seems unrealistic and provides erroneous labels.