Buffer layers for Test-Time Adaptation

TL;DR

The paper tackles domain shift in vision models by criticizing normalization-based test-time adaptation (TTA) for its sensitivity to small batches and reliance on training-time statistics. It proposes a modular Buffer layer that sits in parallel to a frozen backbone, acting as an external, trainable adaptation unit usable with any TTA objective and capable of co-training with normalization when beneficial. Across CIFAR-10/100-C, CIFAR-10-W, and ImageNet-C, Buffer delivers consistent performance gains, strong resilience to catastrophic forgetting, and broad architectural compatibility, even when used with group normalization. The findings suggest a practical, scalable pathway for robust, source-free TTA in dynamic deployment scenarios, with the code openly available for integration into existing pipelines.

Abstract

In recent advancements in Test Time Adaptation (TTA), most existing methodologies focus on updating normalization layers to adapt to the test domain. However, the reliance on normalization-based adaptation presents key challenges. First, normalization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization-based adaptation is inherently constrained by the structure of the pre-trained model, as it relies on training-time statistics that may not generalize well to unseen domains. These issues limit the effectiveness of normalization-based TTA approaches, especially under significant domain shift. In this paper, we introduce a novel paradigm based on the concept of a Buffer layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre-trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods in mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our Buffer layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improvements across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at https://github.com/hyeongyu-kim/Buffer_TTA.

Figures (6)

• Figure 1: Overview of our test-time adaptation framework. Unlike prior methods that rely on updating normalization layers or fine-tuning the entire model, which require backpropagation and can suffer from instability under small batch sizes, or additive modules that require warm-up phases, our proposed Buffer layer enables direct test-time adaptation without any additional training. It operates on any type of objectives to mitigate domain shift, acting as a lightweight and modular adaptation unit that preserves the original model parameters and prevents catastrophic forgetting.
• Figure 2: Classification error rates (%) across datasets on large batch sizes.
• Figure 3: Catastrophic forgetting experiments on WRN28, CIFAR-10-W. Blue: TENT @BN, Red: TENT @Buffer.
• Figure 4: Module design of Buffer layer.
• Figure 5: Catastrophic forgetting experiments on WRN28, CIFAR10-W (KW). Blue: TENT @BN, Red: TENT @Buffer.