Unified Entropy Optimization for Open-Set Test-Time Adaptation

TL;DR

This work tackles open-set test-time adaptation by addressing both covariate and semantic shifts, where test data may include unseen classes. It introduces UniEnt, a distribution-aware framework that first separates csID and csOOD samples via a csOOD score computed from source-domain prototypes, then applies entropy minimization to csID samples and entropy maximization to csOOD samples, with a marginal entropy term to prevent collapse. UniEnt+ adds a sample-weighting scheme to mitigate noise from the initial partition, improving robustness. Empirical results on CIFAR-10-C, CIFAR-100-C, Tiny-ImageNet-C, and larger-scale datasets demonstrate that UniEnt and UniEnt+ consistently outperform state-of-the-art TTA methods in open-set scenarios, achieving better accuracy on known classes and stronger detection of unknowns. The framework is flexible and can augment existing TTA methods, offering a practical path toward robust open-set adaptation in real-world deployments.

Abstract

Test-time adaptation (TTA) aims at adapting a model pre-trained on the labeled source domain to the unlabeled target domain. Existing methods usually focus on improving TTA performance under covariate shifts, while neglecting semantic shifts. In this paper, we delve into a realistic open-set TTA setting where the target domain may contain samples from unknown classes. Many state-of-the-art closed-set TTA methods perform poorly when applied to open-set scenarios, which can be attributed to the inaccurate estimation of data distribution and model confidence. To address these issues, we propose a simple but effective framework called unified entropy optimization (UniEnt), which is capable of simultaneously adapting to covariate-shifted in-distribution (csID) data and detecting covariate-shifted out-of-distribution (csOOD) data. Specifically, UniEnt first mines pseudo-csID and pseudo-csOOD samples from test data, followed by entropy minimization on the pseudo-csID data and entropy maximization on the pseudo-csOOD data. Furthermore, we introduce UniEnt+ to alleviate the noise caused by hard data partition leveraging sample-level confidence. Extensive experiments on CIFAR benchmarks and Tiny-ImageNet-C show the superiority of our framework. The code is available at https://github.com/gaozhengqing/UniEnt

Figures (6)

• Figure 1: Existing TTA methods exhibit performance degradation with unknown classes included, while our methods can improve them significantly. We compare BN Adapt nado2020evaluating, CoTTA wang2022continual, TENT wang2021tent, EATA niu2022efficient, and OSTTA lee2023towards.
• Figure 2: Comparison between closed-set TTA and open-set TTA.
• Figure 3: Illustration of the unified entropy optimization (UniEnt) framework. At timestamp $t$, mini-batch $\mathcal{B}_t$ may contain samples from csID and csOOD. First, we filter csOOD samples by csOOD score $S(\mathbf{x})$. Then, we perform entropy minimization for csID samples and entropy maximization for csOOD samples, we also adopt marginal entropy maximization to pervent model collapse. After optimization, we can yield better classification and detection performance tradeoff.
• Figure 4: Performance comparison of existing TTA methods under closed-set and open-set settings.
• Figure 5: The csOOD score $S(\mathbf{x})$ presents a bimodal distribution.