Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptation

TL;DR

This work tackles multimodal test-time adaptation by addressing modality-specific distribution shifts and cross-modal semantic misalignment. It introduces BriMPR, a two-stage framework that first uses modality-specific prompts to align unimodal global feature distributions (PMFGA) and then refines cross-modal alignment through cross-modal masked embedding recombination and inter-modal instance-wise contrastive learning (CMER+IICL). The approach avoids full fine-tuning, leverages prompt tuning for efficiency, and demonstrates superior performance on corruption, real-world, and continual domain shifts across multiple multimodal benchmarks. The results show BriMPR's strong robustness and effectiveness in bridging modality gaps at test time, with comprehensive ablations validating each component.

Abstract

Test-time adaptation (TTA) enables online model adaptation using only unlabeled test data, aiming to bridge the gap between source and target distributions. However, in multimodal scenarios, varying degrees of distribution shift across different modalities give rise to a complex coupling effect of unimodal shallow feature shift and cross-modal high-level semantic misalignment, posing a major obstacle to extending existing TTA methods to the multimodal field. To address this challenge, we propose a novel multimodal test-time adaptation (MMTTA) framework, termed as Bridging Modalities via Progressive Re-alignment (BriMPR). BriMPR, consisting of two progressively enhanced modules, tackles the coupling effect with a divide-and-conquer strategy. Specifically, we first decompose MMTTA into multiple unimodal feature alignment sub-problems. By leveraging the strong function approximation ability of prompt tuning, we calibrate the unimodal global feature distributions to their respective source distributions, so as to achieve the initial semantic re-alignment across modalities. Subsequently, we assign the credible pseudo-labels to combinations of masked and complete modalities, and introduce inter-modal instance-wise contrastive learning to further enhance the information interaction among modalities and refine the alignment. Extensive experiments on MMTTA tasks, including both corruption-based and real-world domain shift benchmarks, demonstrate the superiority of our method. Our source code is available at https://github.com/Luchicken/BriMPR.

Figures (8)

• Figure 1: t-SNE visualizations of unimodal (top) and fused multimodal (bottom) features during adaptation versus source features. For fused features, 10 classes from Kinetics50–C are shown.
• Figure 2: Overview of BriMPR. BriMPR achieves initial alignment and alignment refinement through two progressive modules. The added modality-specific prompts are used to project the unimodal features into the re-aligned feature space.
• Figure 3: Performance comparison when the data available for adaptation is limited under two tasks on Kinetics50-C and VGGSound-C. (a) and (b) correspond to the unimodal shift setting; (c) and (d) correspond to the multimodal shift setting.
• Figure 4: Comparison with the state-of-the-arts on Kinetics50-C under the continual setting (severity level 5). "Kinetics50-video" contains 15 continuous domains, while "Kinetics50-both" contains $15 \times 6 = 90$ continuous domains. The legend shows the average accuracy across all domains.
• Figure 5: Comparison between updating LN parameters and updating prompts.

Theorems & Definitions (3)

• Theorem 1
• Theorem 1
• proof