SEVA: Leveraging Single-Step Ensemble of Vicinal Augmentations for Test-Time Adaptation

TL;DR

This work tackles distribution-shift robustness at test time by reframing entropy-based self-training to leverage reliable samples through vicinal augmentations. It introduces SEVA, a single-step method that compresses multiple augmentation rounds into one pass via an upper-bound entropy loss termed Augmented Entropy, accompanied by a reliable-sample selection mechanism. The approach is grounded in a theoretical derivation of robust predictions under feature-space Gaussian perturbations and an upper bound $\mathcal{L}_{AE}$ that can be optimized efficiently, plus a threshold-based screening rule using $\mathcal{L}_0$. Empirically, SEVA achieves state-of-the-art results on ImageNet-C across imbalanced label shifts, mixed testing domains, and limited-batch scenarios, while requiring far less runtime than multi-round augmentation methods. This yields practical, real-time TTA with broad applicability to diverse architectures and domain shifts.

Abstract

Test-Time adaptation (TTA) aims to enhance model robustness against distribution shifts through rapid model adaptation during inference. While existing TTA methods often rely on entropy-based unsupervised training and achieve promising results, the common practice of a single round of entropy training is typically unable to adequately utilize reliable samples, hindering adaptation efficiency. In this paper, we discover augmentation strategies can effectively unleash the potential of reliable samples, but the rapidly growing computational cost impedes their real-time application. To address this limitation, we propose a novel TTA approach named Single-step Ensemble of Vicinal Augmentations (SEVA), which can take advantage of data augmentations without increasing the computational burden. Specifically, instead of explicitly utilizing the augmentation strategy to generate new data, SEVA develops a theoretical framework to explore the impacts of multiple augmentations on model adaptation and proposes to optimize an upper bound of the entropy loss to integrate the effects of multiple rounds of augmentation training into a single step. Furthermore, we discover and verify that using the upper bound as the loss is more conducive to the selection mechanism, as it can effectively filter out harmful samples that confuse the model. Combining these two key advantages, the proposed efficient loss and a complementary selection strategy can simultaneously boost the potential of reliable samples and meet the stringent time requirements of TTA. The comprehensive experiments on various network architectures across challenging testing scenarios demonstrate impressive performances and the broad adaptability of SEVA. The code will be publicly available.

Figures (5)

• Figure 1: Multimedia devices often capture corrupted samples, causing distribution shifts that impair model capability. To address this, many TTA methods utilize entropy self-training to adapt the model to these shifts and can use augmented data for multiple training to enhance adaptation, attaining improvements but with significantly increased time overhead. Comparatively, our proposed SEVA achieves the benefits of multi-step augmentation in a single step, yielding impressive results without additional training iterations.
• Figure 2: Performance comparison is conducted on the snow type of ImageNet-C under different training strategies. (a) & (b) record accuracy during online adaptation and compare SAR sar, vicinal augmentation, repeat training, and our SEVA on both ResNet and ViT. "Vicinal/Repeat n Times" denotes utilizing vicinal augmentation/repeat training n times based on SAR. (c) records the final accuracy and compares SEVA with the direct augmentation strategy on four representative corruption types. (d) shows the performance using different augmentation methods.
• Figure 3: Comparison of different approaches for using vicinal augmentation in the TTA scenario: (a) Direct Augmentation: It involves repeatedly sampling augmented features into training from a Gaussian distribution (as the vicinal area). As the number of repetitions $N$ increases, augmented sets become richer, but the computational costs also increase linearly and cannot meet the time requirement. (b) Efficient Approach in SEVA: Instead of directly augmenting samples, we consider the impact of vicinal augmentation on the entropy loss as the number of augmentations approaches infinity. Through theoretical derivation, we obtain a closed-form upper bound to serve as a novel loss $\mathcal{L}_{AE}$ and leverage a single training step of $\mathcal{L}_{AE}$ to achieve a similar effect to multiple rounds in (a), significantly enhancing the adaptation efficiency.
• Figure 4: Comparison of various sample selection methods on the ViT-LN model. A large F1 score indicates that the method can identify reliable samples effectively.
• Figure 5: The left picture shows the performance under different vicinal range $\lambda$ and the right shows the performance under different boundary coefficient $\mathcal{L}_0$.