Enhancing Test Time Adaptation with Few-shot Guidance

TL;DR

The paper tackles domain shift by strengthening Test Time Adaptation with a practical few-shot supervision strategy. It introduces Few-Shot Test Time Adaptation (FS-TTA), a two-stage framework that first aligns the decision boundary via few-shot fine-tuning with a Feature Diversity Augmentation module, then refines distribution with online prototypes and entropy-guided pseudo-labeling. The approach achieves state-of-the-art results on three cross-domain benchmarks, demonstrating robust, privacy-preserving adaptation in streaming target environments. Overall, FS-TTA offers a realistic path to reliable domain adaptation in real-world deployments with limited target annotations.

Abstract

Deep neural networks often encounter significant performance drops while facing with domain shifts between training (source) and test (target) data. To address this issue, Test Time Adaptation (TTA) methods have been proposed to adapt pre-trained source model to handle out-of-distribution streaming target data. Although these methods offer some relief, they lack a reliable mechanism for domain shift correction, which can often be erratic in real-world applications. In response, we develop Few-Shot Test Time Adaptation (FS-TTA), a novel and practical setting that utilizes a few-shot support set on top of TTA. Adhering to the principle of few inputs, big gains, FS-TTA reduces blind exploration in unseen target domains. Furthermore, we propose a two-stage framework to tackle FS-TTA, including (i) fine-tuning the pre-trained source model with few-shot support set, along with using feature diversity augmentation module to avoid overfitting, (ii) implementing test time adaptation based on prototype memory bank guidance to produce high quality pseudo-label for model adaptation. Through extensive experiments on three cross-domain classification benchmarks, we demonstrate the superior performance and reliability of our FS-TTA and framework.

Figures (7)

• Figure 1: Test Time Adaptation (TTA) vs. Few-Shot Test Time Adaptation (FS-TTA). FS-TTA incorporates a small number of labeled target samples, which can be easily collected offline before deployment with minimal annotation effort, in addition to the unlabeled target data used in TTA. The results for TTA are based on the performance of TENT wang2020tent on the OfficeHome venkateswara2017deep.
• Figure 2: Performance comparison across different adaptation strategies on OfficeHome. One-shot fine-tuning with a single labeled sample per class already surpasses TENT, showing the effectiveness of minimal supervision.
• Figure 3: Illustration of our two-stage framework. In Stage I, we employ the few-shot support set to fine-tune the source model. To prevent overfitting, we propose FDA module. In Stage II, we maintain a prototype memory bank to guide test time adaptation. In order to update the prototype memory bank and model with effective samples, we propose the entropy filter and consistency selection modules.
• Figure 4: Comprehensive comparison between our method and the state-of-the-art method in DG/TTA settings on DomainNet.
• Figure 5: (a) Effectiveness of the two-stage framework. Stage I already yields notable gains over the source model, while adding Stage II brings further improvements, demonstrating the complementary roles of boundary alignment and distribution refinement. (b) Effectiveness of the FDA module. FDA improves few-shot fine-tuning over both the baseline and Mix-up, showing that enhanced feature diversity benefits low-shot adaptation.