Dynamic Prompt Allocation and Tuning for Continual Test-Time Adaptation

TL;DR

The paper addresses catastrophic forgetting in continual test-time adaptation by introducing PAINT, a dynamic framework that learns domain-specific prompts stored in a memory and allocated on the fly via a query mechanism. It couples mutual information–based prompt tuning with structural regularization and partial encoder fine-tuning to reduce inter-domain interference while preserving previously learned knowledge. Empirical results on CIFAR10-C, ImageNet-C, and ImageNet-R show state-of-the-art performance, strong anti-forgetting behavior, and robust handling of gradually changing domains. The work highlights the practical potential of domain-specific prompts for CTTA and points to open-set TTA as a promising direction for future research.

Abstract

Continual test-time adaptation (CTTA) has recently emerged to adapt a pre-trained source model to continuously evolving target distributions, which accommodates the dynamic nature of real-world environments. To mitigate the risk of catastrophic forgetting in CTTA, existing methods typically incorporate explicit regularization terms to constrain the variation of model parameters. However, they cannot fundamentally resolve catastrophic forgetting because they rely on a single shared model to adapt across all target domains, which inevitably leads to severe inter-domain interference. In this paper, we introduce learnable domain-specific prompts that guide the model to adapt to corresponding target domains, thereby partially disentangling the parameter space of different domains. In the absence of domain identity for target samples, we propose a novel dynamic Prompt AllocatIon aNd Tuning (PAINT) method, which utilizes a query mechanism to dynamically determine whether the current samples come from a known domain or an unexplored one. For known domains, the corresponding domain-specific prompt is directly selected, while for previously unseen domains, a new prompt is allocated. Prompt tuning is subsequently performed using mutual information maximization along with structural regularization. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our PAINT method for CTTA. We have released our code at https://github.com/Cadezzyr/PAINT.

Figures (6)

• Figure 1: Conceptual framework of our PAINT method for CTTA.
• Figure 2: Images with various corruption types and severity levels on CIFAR10-C and ImageNet-C, as well as some examples in different artistic renditions from ImageNet-R. The images are sourced from hendrycks2019robustness and hendrycks2021many, respectively.
• Figure 3: Anti-forgetting capabilities of PAINT and CoTTA.
• Figure 4: Effects of the prompt length and thresholds.
• Figure 5: Contribution of different components of PAINT.