FRET: Feature Redundancy Elimination for Test Time Adaptation

TL;DR

This work tackles distribution shifts in test-time by focusing on embedding feature redundancy. It introduces FRET, with S-FRET directly minimizing a redundancy score and G-FRET employing a graph-based, attention/redundancy decomposition to enhance both representation and prediction discriminability. Across domain generalization and image corruption benchmarks, S-FRET delivers strong gains and G-FRET achieves state-of-the-art performance, with analyses showing redundancy reduction correlates with improved generalization and robustness to label shift. The approach is demonstrated to be scalable across datasets and architectures, including ViT backbones, making it practical for real-world TTA deployments.

Abstract

Test-Time Adaptation (TTA) aims to enhance the generalization of deep learning models when faced with test data that exhibits distribution shifts from the training data. In this context, only a pre-trained model and unlabeled test data are available, making it particularly relevant for privacy-sensitive applications. In practice, we observe that feature redundancy in embeddings tends to increase as domain shifts intensify in TTA. However, existing TTA methods often overlook this redundancy, which can hinder the model's adaptability to new data. To address this issue, we introduce Feature Redundancy Elimination for Test-time Adaptation (FRET), a novel perspective for TTA. A straightforward approach (S-FRET) is to directly minimize the feature redundancy score as an optimization objective to improve adaptation. Despite its simplicity and effectiveness, S-FRET struggles with label shifts, limiting its robustness in real-world scenarios. To mitigate this limitation, we further propose Graph-based FRET (G-FRET), which integrates a Graph Convolutional Network (GCN) with contrastive learning. This design not only reduces feature redundancy but also enhances feature discriminability in both the representation and prediction layers. Extensive experiments across multiple model architectures, tasks, and datasets demonstrate the effectiveness of S-FRET and show that G-FRET achieves state-of-the-art performance. Further analysis reveals that G-FRET enables the model to extract non-redundant and highly discriminative features during inference, thereby facilitating more robust test-time adaptation.

Figures (6)

• Figure 1: An intuitive demonstration of the relationship between embedded feature redundancy and distribution migration.Top: Second-order feature relation graph of embedding features. Bottom: Plot of feature redundancy versus the level of corruption.
• Figure 2: The pipeline of our proposed Graph Convolutional Network based Feature Redundancy Elimination for Test Time Adaptation (G-FRET) method. For each test sample, G-FRET learns non-redundant and discriminative representation through contrastive distillation in the representation-layer. Additionally, in the prediction layer, attention predictions are optimized through entropy minimization, while ensuring that they are distinct from redundant predictions by using negative learning.
• Figure 3: The relationship of the reduction of redundancy and the enhancement of generalizability attained by S-FRET and G-FRET on CIFAR-10-C and CIFAR-100-C datasets. NRS: Normalized Redundancy Scores.
• Figure 4: Discriminability visualization of embedded feature before and after adaptation by S-FRET and G-FRET respectively.
• Figure 5: Parameter sensitivity analysis G-FRET on the OfficeHome dataset with ResNet-18 backbone in domain Art.