Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization

TL;DR

This work tackles Multimodal Open-set Test-time Adaptation (MM-OSTTA) by showing that enlarging the entropy gap between known and unknown target samples improves online adaptation. It proposes Adaptive Entropy-aware Optimization (AEO), consisting of Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP), to amplify entropy differences and diversify cross-modal predictions during test-time. A new MM-OSTTA benchmark across two tasks and five modalities demonstrates strong improvements over state-of-the-art baselines and resilience in long-term and continual adaptation, including corrupted and mixed-domain shifts. The approach provides practical benefits for robust, real-time multimodal perception systems by maintaining performance while detecting unknown classes at test time.

Abstract

Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on unimodal OSTTA, often filtering out low-confidence samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the ability of model to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM-OSTTA setting, we establish a new benchmark derived from existing datasets. This benchmark includes two downstream tasks and incorporates five modalities. Extensive experiments across various domain shift situations demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTTA settings, both of which are challenging and highly relevant to real-world applications. Our source code is available at https://github.com/donghao51/AEO.

Figures (15)

• Figure 1: (a) Tent minimizes the entropy of all samples, making it difficult to separate the prediction score distributions of known and unknown samples. (b) Our AEO amplifies entropy differences between known and unknown samples through adaptive optimization. (c) As a result, Tent negatively impacts MM-OSTTA performance while AEO significantly improves unknown class detection.
• Figure 2: The entropy difference between known and unknown samples is positively correlated with the MM-OSTTA performance. Tent minimizes the entropy of all samples, regardless of whether they are known or unknown, thereby failing to increase entropy differences and leading to poorer performances. In contrast, our AEO amplifies entropy differences via adaptive optimization, significantly improving unknown detection. Different shapes represent different domain-shift scenarios.
• Figure 3: Training with AMP further amplifies the entropy difference between known and unknown samples, leading to improved performances.
• Figure 4: AEO continuously optimizes entropy difference between known and unknown samples, resulting in a substantial reduction of FPR95 after $10$ adaptation epochs.
• Figure 5: Parameter sensitivity analysis using video and audio modalities on the HAC dataset.