Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation

TL;DR

This work tackles Fully Test-Time Adaptation (FTTA) under the practical constraint of no access to source data or training protocols. It introduces Agnostic FTTA (AFTTA) and a concrete implementation, TIRNU, which uncovers plausible nuisance shifts via off-the-shelf augmentations and then unlearns their influence through a mutual-information objective in the feature space, complemented by a label-consistency regularizer. A variational perspective motivates the approach, modeling domain shifts as latent nuisances and employing a surrogate distribution over transformations for robust prediction. Empirical results across diverse domain shifts—corruptions, natural/adversarial changes, and style transfer—demonstrate that TIRNU outperforms strong baselines and achieves state-of-the-art or competitive performance, albeit with higher computational cost due to augmentation and MI estimation. The method offers a flexible framework for incorporating domain-shift priors into FTTA, opening avenues for tailored nuisance priors and broader applications in domain adaptation under data access constraints.

Abstract

Fully Test-Time Adaptation (FTTA) addresses domain shifts without access to source data and training protocols of the pre-trained models. Traditional strategies that align source and target feature distributions are infeasible in FTTA due to the absence of training data and unpredictable target domains. In this work, we exploit a dual perspective on FTTA, and propose Agnostic FTTA (AFTTA) as a novel formulation that enables the usage of off-the-shelf domain transformations during test-time to enable direct generalization to unforeseeable target data. To address this, we develop an uncover-and-unlearn approach. First, we uncover potential unwanted shifts between source and target domains by simulating them through predefined mappings and consider them as nuisances. Then, during test-time prediction, the model is enforced to unlearn these nuisances by regularizing the consequent shifts in latent representations and label predictions. Specifically, a mutual information-based criterion is devised and applied to guide nuisances unlearning in the feature space and encourage confident and consistent prediction in label space. Our proposed approach explicitly addresses agnostic domain shifts, enabling superior model generalization under FTTA constraints. Extensive experiments on various tasks, involving corruption and style shifts, demonstrate that our method consistently outperforms existing approaches.

Figures (4)

• Figure 1: Illustration of Agnostic Fully Test-Time Adaptation (AFTTA).
• Figure 2: Our TIRNU approach for agnostic fully test-time domain adaptation. Without access to the (source) training data, we adopt an uncover-and-unlearn strategy. We uncover potential unwanted shifts between the source and target as nuisances and mimic these nuisances onto the target data with off-the-shelf data transformations. During test-time adaptation, the feature extractor is updated to unlearn these nuisance shifts by mitigating their impact on feature representations, ensuring confident and consistent label predictions. Our model enhances the source model’s generalization to the agnostic target domain by promoting shift-invariant feature representations.
• Figure 3: t-SNE visualization of features extracted by MEMO and TIRNU on CIFAR10-C.
• Figure 4: Average error (%) for the CIFAR10-C dataset for (left) different values of $\alpha$, and (right) different values of $k$. The model performance is not sensitive to either of these two hyperparameters.