Discover Your Neighbors: Advanced Stable Test-Time Adaptation in Dynamic World

TL;DR

The paper tackles performance degradation under distribution shifts by proposing Discover Your Neighbors (DYN), a backward-free test-time adaptation method designed for dynamic data streams. DYN combines Layer-wise Instance Statistics Clustering (LISC) with Cluster-Aware Batch Normalization (CABN) to iteratively group samples with similar feature distributions and fuse cluster-specific statistics with source BN statistics, improving robustness to mixed-domain batches. The work analyzes the roles of SBN and TBN in capturing class-related and class-irrelevant features, revealing that a careful, cluster-aware integration of BN statistics yields more stable representations in dynamic settings. Experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C show that DYN outperforms state-of-the-art TTA and TBN-based methods across multiple scenarios, model architectures, domain scales, and batch sizes, highlighting its practical impact for real-time multimedia applications.

Abstract

Despite progress, deep neural networks still suffer performance declines under distribution shifts between training and test domains, leading to a substantial decrease in Quality of Experience (QoE) for multimedia applications. Existing test-time adaptation (TTA) methods are challenged by dynamic, multiple test distributions within batches. This work provides a new perspective on analyzing batch normalization techniques through class-related and class-irrelevant features, our observations reveal combining source and test batch normalization statistics robustly characterizes target distributions. However, test statistics must have high similarity. We thus propose Discover Your Neighbours (DYN), the first backward-free approach specialized for dynamic TTA. The core innovation is identifying similar samples via instance normalization statistics and clustering into groups which provides consistent class-irrelevant representations. Specifically, Our DYN consists of layer-wise instance statistics clustering (LISC) and cluster-aware batch normalization (CABN). In LISC, we perform layer-wise clustering of approximate feature samples at each BN layer by calculating the cosine similarity of instance normalization statistics across the batch. CABN then aggregates SBN and TCN statistics to collaboratively characterize the target distribution, enabling more robust representations. Experimental results validate DYN's robustness and effectiveness, demonstrating maintained performance under dynamic data stream patterns.

Figures (7)

• Figure 1: Top: illustration of test-time adaptation (TTA). Bottom: Comparison with different TTA methods. The proposed DYN can adjust the characterization of target domain feature distributions through LISC and CABN, enabling the model to maintain high performance and robustness under dynamic conditions
• Figure 2: TBN-IN distance vs. accuracy under different corruptions. The 5 data points in the figure represent samples with corruption levels 1 through 5, where higher levels correspond to lower accuracy. We compute the average distance between per-sample IN statistics and TBN statistics in the deep layers, reflecting the dispersion of feature distributions within a batch. It reveals that the distribution of CIF interferes with the distribution of CRF: as the sample corruption level increases, the feature distributions within a batch become more coupled.
• Figure 3: (a) shows the inference accuracy of the pre-trained model after splitting images into varying numbers of patches and shuffling them. As the image is divided into smaller patches, feature reuse becomes more difficult for the model, resulting in lower inference accuracy. (b) displays the model's inference accuracy on samples with different corruptions versus the dispersion of per-sample IN statistics within a batch. Each data point represents samples from a distinct corruption type (level 5). (c) and (d) show the model's inference accuracy when combining TBN with SBN and when combining IN with SBN, respectively. In both figures, $\alpha$ denotes the proportion of SBN statistics used in the combination.
• Figure 4: Patch visualization. As the image is cropped into an increasing number of patches, the class-relevant features become harder to capture, while the class-irrelevant features are still retained. In this case, the inference accuracy of the model rapidly decline.
• Figure 5: Airplane wings in different corruption. More severe corruption perturbations of the image can obscure the class-relevant features, causing the overall feature distribution to become more entangled and thus decreasing the model's inference accuracy.