Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation

Chaoqun Du, Yulin Wang, Jiayi Guo, Yizeng Han, Jie Zhou, Gao Huang

TL;DR

This work introduces UniTTA, a unified benchmark and versatile framework for Realistic Test-Time Adaptation (TTA). It defines 36 scenarios by modeling domain and class shifts with Markov state transitions (ULMMs) and combines them via Kronecker products, enabling realistic i.i.d., non-i.i.d., continual, and imbalanced/balanced conditions. The UniTTA framework comprises Balanced Domain Normalization (BDN) to obtain domain-balanced statistics and COFA to exploit temporal feature correlations, with a confidence-based filter to maintain performance in i.i.d. settings. Extensive experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C show state-of-the-art average performance across 24 realistic settings, illustrating robustness and practical value for practitioners selecting TTA methods. The work provides code and a scalable, extensible paradigm for evaluating and deploying robust TTA in real-world deployments.

Abstract

Test-Time Adaptation (TTA) aims to adapt pre-trained models to the target domain during testing. In reality, this adaptability can be influenced by multiple factors. Researchers have identified various challenging scenarios and developed diverse methods to address these challenges, such as dealing with continual domain shifts, mixed domains, and temporally correlated or imbalanced class distributions. Despite these efforts, a unified and comprehensive benchmark has yet to be established. To this end, we propose a Unified Test-Time Adaptation (UniTTA) benchmark, which is comprehensive and widely applicable. Each scenario within the benchmark is fully described by a Markov state transition matrix for sampling from the original dataset. The UniTTA benchmark considers both domain and class as two independent dimensions of data and addresses various combinations of imbalance/balance and i.i.d./non-i.i.d./continual conditions, covering a total of \( (2 \times 3)^2 = 36 \) scenarios. It establishes a comprehensive evaluation benchmark for realistic TTA and provides a guideline for practitioners to select the most suitable TTA method. Alongside this benchmark, we propose a versatile UniTTA framework, which includes a Balanced Domain Normalization (BDN) layer and a COrrelated Feature Adaptation (COFA) method--designed to mitigate distribution gaps in domain and class, respectively. Extensive experiments demonstrate that our UniTTA framework excels within the UniTTA benchmark and achieves state-of-the-art performance on average. Our code is available at \url{https://github.com/LeapLabTHU/UniTTA}.

UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation

TL;DR

This work introduces UniTTA, a unified benchmark and versatile framework for Realistic Test-Time Adaptation (TTA). It defines 36 scenarios by modeling domain and class shifts with Markov state transitions (ULMMs) and combines them via Kronecker products, enabling realistic i.i.d., non-i.i.d., continual, and imbalanced/balanced conditions. The UniTTA framework comprises Balanced Domain Normalization (BDN) to obtain domain-balanced statistics and COFA to exploit temporal feature correlations, with a confidence-based filter to maintain performance in i.i.d. settings. Extensive experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C show state-of-the-art average performance across 24 realistic settings, illustrating robustness and practical value for practitioners selecting TTA methods. The work provides code and a scalable, extensible paradigm for evaluating and deploying robust TTA in real-world deployments.

Abstract

Test-Time Adaptation (TTA) aims to adapt pre-trained models to the target domain during testing. In reality, this adaptability can be influenced by multiple factors. Researchers have identified various challenging scenarios and developed diverse methods to address these challenges, such as dealing with continual domain shifts, mixed domains, and temporally correlated or imbalanced class distributions. Despite these efforts, a unified and comprehensive benchmark has yet to be established. To this end, we propose a Unified Test-Time Adaptation (UniTTA) benchmark, which is comprehensive and widely applicable. Each scenario within the benchmark is fully described by a Markov state transition matrix for sampling from the original dataset. The UniTTA benchmark considers both domain and class as two independent dimensions of data and addresses various combinations of imbalance/balance and i.i.d./non-i.i.d./continual conditions, covering a total of \( (2 \times 3)^2 = 36 \) scenarios. It establishes a comprehensive evaluation benchmark for realistic TTA and provides a guideline for practitioners to select the most suitable TTA method. Alongside this benchmark, we propose a versatile UniTTA framework, which includes a Balanced Domain Normalization (BDN) layer and a COrrelated Feature Adaptation (COFA) method--designed to mitigate distribution gaps in domain and class, respectively. Extensive experiments demonstrate that our UniTTA framework excels within the UniTTA benchmark and achieves state-of-the-art performance on average. Our code is available at \url{https://github.com/LeapLabTHU/UniTTA}.
Paper Structure (30 sections, 2 theorems, 15 equations, 6 figures, 12 tables)

This paper contains 30 sections, 2 theorems, 15 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark
  4. Existing Realistic TTA Settings
  5. UniTTA Benchmark
  6. Sampling Time.
  7. Discussion
  8. UniTTA Framework
  9. Overview
  10. Balanced Domain Normalization
  11. Domain assignment (prediction) and expansion.
  12. Domain-class statistics update.
  13. COFA
  14. Experiments
  15. Results
  16. ...and 15 more sections

Key Result

Proposition 1

For a Uniformly Leaving Markov Matrix with diagonal elements $\bm{\alpha}$ where $\alpha_i = P_{ii}$ for all $i$, there exists a unique stationary distribution $\bm{\pi} = (\pi_1, \pi_2, \cdots, \pi_n)$. This distribution satisfies the following relationship:

Figures (6)

  • Figure 1: Data generation process for the UniTTA benchmark. Continual TTA describes a scenario in which the domain remains consistent over an extended period before shifting to a new domain, which exemplifies an extreme case of non-i.i.d. settings. We consider the domain and class as two independent attributes, each associated with its own Markov matrix.
  • Figure 2: The overall architecture of the UniTTA framework. The original model's BN layers are replaced by BDN layers, and the linear classifier is equipped with the COFA method. The UniTTA framework sequentially predicts the class label and domain label in the $m$-th BDN layer through three forward passes, ultimately providing the final prediction.
  • Figure 3: Average error (%) on CIFAR10-C under various correlation and imbalance factors. The default factors for domain and class are (0.85, 5) and (0.95, 10), respectively. In two sets of experiments, we kept either the domain or class factors constant while varying the other.
  • Figure 4: Sensitive analysis of batch size on CIFAR10-C. The default correlation and imbalance factors for domain and class are $(0.85, 5)$ and $(0.95, 10)$, rspectively.
  • Figure 5: Sensitivity analysis of the BDN layer for domain prediction on CIFAR100-C. The horizontal axis $(m,n)$ indicates the $n$th layer of the $m$th block in the network.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Definition 1: Uniformly Leaving Markov Matrix
  • Proposition 1: Stationary Distribution
  • Corollary 1: Temporal Correlation and Imbalance
  • proof : Proof of \ref{['prop:stationary']}