FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions

TL;DR

FREST tackles semantic segmentation under multiple adverse conditions in a source-free domain adaptation setting by alternating between learning a condition embedding space and restoring adverse-condition features to resemble normal-condition representations in that space. It introduces a light-weight condition strainer attached to a frozen encoder and a projection head to capture condition-specific information, and a feature-restoration objective that aligns adverse features with normal-condition embeddings while discriminating between encoder and condition-infused features. This approach reduces condition-induced feature gaps, avoids catastrophic forgetting of source-domain knowledge, and achieves state-of-the-art results on Cityscapes→ACDC and Cityscapes→RobotCar, with strong generalization to unseen domains. The method is parameter-efficient, requires no source data during adaptation, and relies on publicly available GNSS-based pairs and warping for alignment, offering practical utility for real-world robust semantic segmentation.

Abstract

Robust semantic segmentation under adverse conditions is crucial in real-world applications. To address this challenging task in practical scenarios where labeled normal condition images are not accessible in training, we propose FREST, a novel feature restoration framework for source-free domain adaptation (SFDA) of semantic segmentation to adverse conditions. FREST alternates two steps: (1) learning the condition embedding space that only separates the condition information from the features and (2) restoring features of adverse condition images on the learned condition embedding space. By alternating these two steps, FREST gradually restores features where the effect of adverse conditions is reduced. FREST achieved a state of the art on two public benchmarks (i.e., ACDC and RobotCar) for SFDA to adverse conditions. Moreover, it shows superior generalization ability on unseen datasets.

Figures (13)

• Figure 1: (a) The setting of SFDA to adverse conditions. A segmentation model, initially pre-trained on a labeled source dataset, is adapted to adverse conditions using pairs of unlabeled adverse and normal images. (b) Following the SFDA setting, FREST restores features of adverse condition images to simulate the normal condition.
• Figure 2: The overall architecture and training strategy. The segmentation network is pre-trained using a labeled source dataset. For each iteration, the condition strainer and segmentation network are trained alternatingly. The frozen modules are shown in gray, the trainable modules are highlighted in red, and "sg" denotes the stop gradient. (Step 1) The condition strainer and projection head are trained to learn the condition embedding space. (Step 2) The segmentation network is trained to restore features from adverse to normal conditions on the condition embedding space. For evaluation, only the encoder $\phi_\textrm{enc}$ and decoder $\phi_\textrm{dec}$ of the segmentation network are utilized.
• Figure 3: Detail of the encoder with the condition strainer. Condition strainers are connected to the original feed-forward layer (FFN) and multi-head self-attention layer (MHSA) through the residual connections.
• Figure 4: Detail of positive embedding sampling strategy in the condition-specific learning.
• Figure 5: Detail of the adverse condition discriminator.