Shared & Domain Self-Adaptive Experts with Frequency-Aware Discrimination for Continual Test-Time Adaptation

TL;DR

The paper tackles Continual Test-Time Adaptation (CTTA) by addressing the trade-off between rapid adaptation and forgetting across evolving and recurring target domains. It introduces a frequency-aware shared and self-adaptive domain-expert framework with a dual-branch architecture, comprising a domain-shared MoE and an expandable pool of domain self-adaptive MoEs, guided by an online Frequency-Aware Domain Discriminator (FDD) that uses low-frequency features and a Bayesian posterior with a shrinkage-regularized Mahalanobis distance $m_i(z)$ to route inputs and detect new domains. A residual fusion combines general and domain-specific outputs, with online updates to domain statistics and dynamic creation of domain experts, achieving efficient, robust CTTA. Evaluations on classification and semantic segmentation benchmarks, including the Continual Repeated Shifts (CRS) setting, show state-of-the-art performance and reduced forgetting, highlighting the method's practicality for non-stationary real-world environments. The work provides a solid framework for real-time adaptation with memory-efficient domain specialization and introduces CRS to better simulate periodic environmental changes.

Abstract

This paper focuses on the Continual Test-Time Adaptation (CTTA) task, aiming to enable an agent to continuously adapt to evolving target domains while retaining previously acquired domain knowledge for effective reuse when those domains reappear. Existing shared-parameter paradigms struggle to balance adaptation and forgetting, leading to decreased efficiency and stability. To address this, we propose a frequency-aware shared and self-adaptive expert framework, consisting of two key components: (i) a dual-branch expert architecture that extracts general features and dynamically models domain-specific representations, effectively reducing cross-domain interference and repetitive learning cost; and (ii) an online Frequency-aware Domain Discriminator (FDD), which leverages the robustness of low-frequency image signals for online domain shift detection, guiding dynamic allocation of expert resources for more stable and realistic adaptation. Additionally, we introduce a Continual Repeated Shifts (CRS) benchmark to simulate periodic domain changes for more realistic evaluation. Experimental results show that our method consistently outperforms existing approaches on both classification and segmentation CTTA tasks under standard and CRS settings, with ablations and visualizations confirming its effectiveness and robustness. Our code is available at https://github.com/ZJC25127/Domain-Self-Adaptive-CTTA.git.

Figures (4)

• Figure 1: A New Benchmark and Comparison of Methods for CTTA. (a) Our proposed CRS (Continual Repeated Shifts) benchmark simulates realistic, cyclic domain shifts at test time. (b) Existing shared-parameter methods struggle to balance adaptation and memory, leading to forgetting and redundant relearning when domains recur. (c) Our method adds domain self-adaptive experts, enabling knowledge retention and fast adaptation to recurring domains.
• Figure 1: Visualization of the domain-shared expert and the three domain self-adaptive experts on three corruption domains (Gaussian, Snow, and Pixelate). Each panel is an Attention Map, obtained by measuring the similarity between the [CLS] token and all patch tokens produced by the corresponding expert branch; brighter colours denote stronger affinity. The column labelled “Activated’’ indicates that the expert was selected by the Frequency-aware Domain Discriminator (FDD) for the current input, whereas the remaining experts were inactive.
• Figure 2: Overall framework. (a) The Dual-Branch Expert Architecture combines a shared expert with a dynamically expandable set of domain self-adaptive experts. The appropriate domain self-adaptive expert is selected based on input, guided by the online Frequency-aware Domain Discriminator (FDD). (b) FDD detects domain shifts using low-frequency features from 2D Fourier-transformed inputs, updating domain statistics or initializing new experts when unseen domains are encountered.
• Figure 3: FDD visualization and feature ablation on ImageNet‑C. (a) PCA projection of low‑frequency features with full FDD; six well‑separated clusters are observed, “$\times$” denote estimated domain centres $\mu_i$. (b) Same projection with FDD without covariance, showing overlapping clusters and reduced separability. (c) Feature-Selection Ablation showing classification accuracy on ImageNet-C when the default low-frequency descriptor is replaced with four alternatives.