Adaptive Cascading Network for Continual Test-Time Adaptation

TL;DR

This work proposes a cascading paradigm that simultaneously updates the feature extractor and classifier at test time, mitigating the mismatch between them and enabling long-term model adaptation.

Abstract

We study the problem of continual test-time adaption where the goal is to adapt a source pre-trained model to a sequence of unlabelled target domains at test time. Existing methods on test-time training suffer from several limitations: (1) Mismatch between the feature extractor and classifier; (2) Interference between the main and self-supervised tasks; (3) Lack of the ability to quickly adapt to the current distribution. In light of these challenges, we propose a cascading paradigm that simultaneously updates the feature extractor and classifier at test time, mitigating the mismatch between them and enabling long-term model adaptation. The pre-training of our model is structured within a meta-learning framework, thereby minimizing the interference between the main and self-supervised tasks and encouraging fast adaptation in the presence of limited unlabelled data. Additionally, we introduce innovative evaluation metrics, average accuracy and forward transfer, to effectively measure the model's adaptation capabilities in dynamic, real-world scenarios. Extensive experiments and ablation studies demonstrate the superiority of our approach in a range of tasks including image classification, text classification, and speech recognition.

Figures (4)

• Figure 1: (a) Parallel paradigm for test-time training sun2019test, and (b) our proposed cascading paradigm for continual test-time training. The proposed cascading paradigm efficiently mitigates the mismatch between the feature extractor and main classifier, enabling long-term model adaptation.
• Figure 2: Overview of pre-training and adaptation phases. $\pi \sim \Psi$ denotes a transformation randomly sampled from a predefined pool of transformations. $\mathcal{L}_{\text{ENT}}$: entropy loss, $\mathcal{L}_{\text{CE}}$: cross-entropy loss.
• Figure 3: Validation of the cascading paradigm on CIFAR-10-C. Left: Classification error of models w/ and w/o the auxiliary classifier $\theta_a$. Right: Classification error of models updating and fixing the main classifier $\theta_m$ at test time.
• Figure 4: Uncertainty quantification on CIFAR-10-C. Entropy and error after adapting to different numbers of test samples (left) and different levels of corruption severity (right). Entropy shows consistent trends with the classification error.