Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation

TL;DR

The paper tackles CTTA for semantic segmentation under continual domain shifts, where updating many parameters risks error accumulation and catastrophic forgetting. It introduces Distribution-Aware Tuning (DAT), a parameter-efficient approach that updates two small parameter groups—Domain-Specific Parameters (DSP) and Task-Relevant Parameters (TRP)—guided by pixel-wise distribution shifts detected via an uncertainty map within a mean-teacher framework; a Parameter Accumulation Update (PAU) strategy aggregates updates over target-domain sequences. Empirical results on Cityscapes-to-ACDC and SHIFT show that DAT yields competitive or superior performance with limited parameter updates, while reducing forgetting compared with prior methods. The approach offers a practical path toward robust, real-world CTTA for autonomous driving by balancing adaptation speed, memory efficiency, and prediction stability.

Abstract

Since autonomous driving systems usually face dynamic and ever-changing environments, continual test-time adaptation (CTTA) has been proposed as a strategy for transferring deployed models to continually changing target domains. However, the pursuit of long-term adaptation often introduces catastrophic forgetting and error accumulation problems, which impede the practical implementation of CTTA in the real world. Recently, existing CTTA methods mainly focus on utilizing a majority of parameters to fit target domain knowledge through self-training. Unfortunately, these approaches often amplify the challenge of error accumulation due to noisy pseudo-labels, and pose practical limitations stemming from the heavy computational costs associated with entire model updates. In this paper, we propose a distribution-aware tuning (DAT) method to make the semantic segmentation CTTA efficient and practical in real-world applications. DAT adaptively selects and updates two small groups of trainable parameters based on data distribution during the continual adaptation process, including domain-specific parameters (DSP) and task-relevant parameters (TRP). Specifically, DSP exhibits sensitivity to outputs with substantial distribution shifts, effectively mitigating the problem of error accumulation. In contrast, TRP are allocated to positions that are responsive to outputs with minor distribution shifts, which are fine-tuned to avoid the catastrophic forgetting problem. In addition, since CTTA is a temporal task, we introduce the Parameter Accumulation Update (PAU) strategy to collect the updated DSP and TRP in target domain sequences. We conduct extensive experiments on two widely-used semantic segmentation CTTA benchmarks, achieving promising performance compared to previous state-of-the-art methods.

Figures (4)

• Figure 1: The CTTA problem aims to adapt the source model to continually changing target domains. Existing model-based methods focus on utilizing a majority of parameters to fit different target domain knowledge in continually changing environments. Differently, we propose a parameter-efficient model-based method named distribution-aware tuning (DAT), a novel paradigm for stable continual adaptation. DAT adaptively selects two small groups (e.g., 5%) of trainable parameters based on the data distribution, enabling the simultaneous extraction of domain-specific and task-relevant knowledge. Instead of updating the entire model, DAT streamlines the continual adaptation process, enhancing its efficiency and potential for real-world applications. The red lines indicate the updated parameters.
• Figure 2: The overall framework of distribution-aware tuning. (a) In our approach, we implement a teacher-student framework to predict semantic segmentation maps, utilizing a pixel-wise consistency loss for optimization. Simultaneously, we obtain an uncertainty map from the teacher model to evaluate pixel-level distribution shifts. Based on the degree of these shifts, we selectively choose two small groups of trainable parameters (e.g., 5.0%), namely domain-specific parameters (DSP) and task-relevant parameters (TRP). (b) In the process of DSP and TRP selection, we employ the Parameter Accumulation Update (PAU) strategy to gather these parameters. When processing a series of target domain samples, we opt to select and update only a minute fraction of parameters (e.g., 0.1%) for each sample, subsequently adding these parameters to the selected parameter group. (c) We show the details of DSP and TRP selection, which involves identifying the most sensitive parameters for pixels with significant and minor distribution shifts, respectively.
• Figure 3: Qualitative comparisons on the ACDC dataset. Our method could better segment different pixel-wise classes such as shown in the white box.
• Figure 4: Different percentages of updated parameters (DSP + TRP)