Controllable Continual Test-Time Adaptation

TL;DR

The paper tackles continual test-time adaptation under continual, unlabeled domain shifts by shifting from suppression to guidance of domain shifts. It introduces C-CoTTA, which uses Concept Activation Vectors (CAV) to represent shift directions via prototypes and enforces domain-level (CDS) and class-level (CCS) shift controls within a mean-teacher framework and symmetric cross-entropy loss. Key contributions include a prototype-based shift representation, two explicit controllability losses, and extensive experiments across CIFAR10-C, CIFAR100-C, and ImageNet-C demonstrating improved robustness, reduced category bleeding, and favorable long-term behavior. This approach advances CTTA by preserving class boundaries and reducing sensitivity to domain transformations, with practical implications for robust long-term deployment in dynamic environments.

Abstract

Continual Test-Time Adaptation (CTTA) is an emerging and challenging task where a model trained in a source domain must adapt to continuously changing conditions during testing, without access to the original source data. CTTA is prone to error accumulation due to uncontrollable domain shifts, leading to blurred decision boundaries between categories. Existing CTTA methods primarily focus on suppressing domain shifts, which proves inadequate during the unsupervised test phase. In contrast, we introduce a novel approach that guides rather than suppresses these shifts. Specifically, we propose $\textbf{C}$ontrollable $\textbf{Co}$ntinual $\textbf{T}$est-$\textbf{T}$ime $\textbf{A}$daptation (C-CoTTA), which explicitly prevents any single category from encroaching on others, thereby mitigating the mutual influence between categories caused by uncontrollable shifts. Moreover, our method reduces the sensitivity of model to domain transformations, thereby minimizing the magnitude of category shifts. Extensive quantitative experiments demonstrate the effectiveness of our method, while qualitative analyses, such as t-SNE plots, confirm the theoretical validity of our approach.

Figures (8)

• Figure 1: t-SNE visualization of controllable domain shift in CTTA. (a) For CoTTA wang2022continual, due to the lack of control over domain shift, categories being biased towards others, resulting in fuzzy classification boundaries. (b) In contrast, our method achieves controllable domain shift, so even if categories are shift, it will not lead to confusion among categories.
• Figure 2: The pipeline of C-CoTTA. (a) Based on the mean teacher framework, perturb the student to enhance model robustness, while optimizing using symmetric cross-entropy. (b) Control the overall domain shift by constraining the model's sensitivity to domain shift directions. (c) Control the shift of a specific category by directly controlling the shift direction of any category to prevent bias towards other categories.
• Figure 3: Visualization of the t-SNE dimensionality reduction of three classes from CIFAR10-C dataset (three easily misclassified animals: bird, deer, frog) transferred from the source domain to the target domain (zoom)
• Figure 4: (a) Inter-class distance: can indicate the separability between classes. (b) Inter-domain distance: can indicate the sensitivity of the model to domain transformations.
• Figure 5: Results for Gradual Adaptation