Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification

TL;DR

TAHCD tackles reliable multimodal classification under heterogeneous noise by jointly removing modality-specific and cross-modality noise at global (ASSA) and instance (SACA) levels. It introduces a test-time cooperative enhancement (TTCE) that uses instance-level noise to adapt global denoising, enabling robust performance under unseen noise. The framework relies on covariance-based stable subspace masking with inter-class orthogonality and cross-modality projection alignment, and confidence-aware slack alignment guided by priors estimated from denoised features. Experiments on BRCA, ROSMAP, CUB, and FOOD101 demonstrate superior robustness and generalization compared to state-of-the-art methods, with ablations confirming the contributions of ASSA, SACA, and TTCE to performance gains and adaptation capability.

Abstract

Reliable learning on low-quality multimodal data is a widely concerning issue, especially in safety-critical applications. However, multimodal noise poses a major challenge in this domain and leads existing methods to suffer from two key limitations. First, they struggle to reliably remove heterogeneous data noise, hindering robust multimodal representation learning. Second, they exhibit limited adaptability and generalization when encountering previously unseen noise. To address these issues, we propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD). On one hand, TAHCD introduces the Adaptive Stable Subspace Alignment and Sample-Adaptive Confidence Alignment to reliably remove heterogeneous noise. They account for noise at both global and instance levels and enable jointly removal of modality-specific and cross-modality noise, achieving robust learning. On the other hand, TAHCD introduces test-time cooperative enhancement, which adaptively updates the model in response to input noise in a label-free manner, improving adaptability and generalization. This is achieved by collaboratively enhancing the joint removal process of modality-specific and cross-modality noise across global and instance levels according to sample noise. Experiments on multiple benchmarks demonstrate that the proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable multimodal learning approaches.

Figures (9)

• Figure 1: Illustration of complex multimodal noise in real-world scenario. (i) Heterogeneity. Multimodal noise generally falls into two types zhang2024multimodal: modality-specific and cross-modality noise, which may occur at the global level across all samples or at the instance level in individual samples. (ii) Unpredictability. With incremental data acquisition, the noise evolves over time and may include previously unseen patterns.
• Figure 2: The framework diagram of the Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD), best viewed in color. TAHCD consists of three key components: (1) Adaptive Stable Subspace Alignment (ASSA): jointly removes modality-specific and cross-modality noise at the global level. (2) Sample-Adaptive Confidence Alignment (SACA): jointly removes modality-specific and cross-modality noise at the instance level. (3) Test-Time Cooperative Enhancement (TTCE): adaptively enhances the model in response to test-time noise. Without loss of generality, the diagram illustrates a scenario with two modalities, where the blue and green colors represent two distinct modalities.
• Figure 3: Visualization of masks $w^m_{\lambda}$ learnd by ASSA on CUB image features, before and after adding noise ($\epsilon=5$) to half of the feature dimensions.
• Figure 4: T-SNE visualization of the learned modality representations and the value of cross-modality alignment loss $\mathcal{L}_{\mathrm{nll}}^c$ results on the CUB dataset with both modalities corrupted by modality-specific and cross-modality noise ($\epsilon=5, \eta=10\%$) after different TTCE co-enhancement iterations. Points in different colors represent sample representations from different classes.
• Figure 5: Visualization of sample-wise masks learned by the modality-specific noise expert on the corrupted image modality of CUB, before and after adding modality-specific noise ($\epsilon=5$).