Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Representation Learning by Alternating Unimodal Adaptation

Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, Huaxiu Yao

TL;DR

Modality laziness in multimodal learning arises when dominant modalities overshadow others. The paper proposes Multimodal Learning with Alternating Unimodal Adaptation (MLA), which alternates unimodal optimization with a shared head $g$ and employs a gradient modification matrix $P_t$ to prevent cross-modal forgetting, while using test-time uncertainty-based fusion to integrate modalities. This approach improves cross-modal interactions without requiring paired multimodal data during training and demonstrates strong gains across five datasets in both complete and missing modality scenarios. The work also analyzes modality isolation, expands modality-gap understanding, and shows robustness with pre-trained encoders like CLIP, highlighting practical impact for robust, flexible multimodal learning.

Abstract

Multimodal learning, which integrates data from diverse sensory modes, plays a pivotal role in artificial intelligence. However, existing multimodal learning methods often struggle with challenges where some modalities appear more dominant than others during multimodal learning, resulting in suboptimal performance. To address this challenge, we propose MLA (Multimodal Learning with Alternating Unimodal Adaptation). MLA reframes the conventional joint multimodal learning process by transforming it into an alternating unimodal learning process, thereby minimizing interference between modalities. Simultaneously, it captures cross-modal interactions through a shared head, which undergoes continuous optimization across different modalities. This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information. During the inference phase, MLA utilizes a test-time uncertainty-based model fusion mechanism to integrate multimodal information. Extensive experiments are conducted on five diverse datasets, encompassing scenarios with complete modalities and scenarios with missing modalities. These experiments demonstrate the superiority of MLA over competing prior approaches. Our code is available at https://github.com/Cecile-hi/Multimodal-Learning-with-Alternating-Unimodal-Adaptation.

Multimodal Representation Learning by Alternating Unimodal Adaptation

TL;DR

Modality laziness in multimodal learning arises when dominant modalities overshadow others. The paper proposes Multimodal Learning with Alternating Unimodal Adaptation (MLA), which alternates unimodal optimization with a shared head and employs a gradient modification matrix to prevent cross-modal forgetting, while using test-time uncertainty-based fusion to integrate modalities. This approach improves cross-modal interactions without requiring paired multimodal data during training and demonstrates strong gains across five datasets in both complete and missing modality scenarios. The work also analyzes modality isolation, expands modality-gap understanding, and shows robustness with pre-trained encoders like CLIP, highlighting practical impact for robust, flexible multimodal learning.

Abstract

Multimodal learning, which integrates data from diverse sensory modes, plays a pivotal role in artificial intelligence. However, existing multimodal learning methods often struggle with challenges where some modalities appear more dominant than others during multimodal learning, resulting in suboptimal performance. To address this challenge, we propose MLA (Multimodal Learning with Alternating Unimodal Adaptation). MLA reframes the conventional joint multimodal learning process by transforming it into an alternating unimodal learning process, thereby minimizing interference between modalities. Simultaneously, it captures cross-modal interactions through a shared head, which undergoes continuous optimization across different modalities. This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information. During the inference phase, MLA utilizes a test-time uncertainty-based model fusion mechanism to integrate multimodal information. Extensive experiments are conducted on five diverse datasets, encompassing scenarios with complete modalities and scenarios with missing modalities. These experiments demonstrate the superiority of MLA over competing prior approaches. Our code is available at https://github.com/Cecile-hi/Multimodal-Learning-with-Alternating-Unimodal-Adaptation.
Paper Structure (48 sections, 8 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 48 sections, 8 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Imbalanced Modality Contributions to Multimodal Learning
  4. Learning with Missing Modalities
  5. Multimodal Learning with Alternating Unimodal Adaptation
  6. Alternating Unimodal Learning
  7. Learning Cross-Modal Information without Modality Forgetting
  8. Test-Time Dynamic Modality Fusion
  9. Experiments
  10. Learning with complete modalities
  11. Experimental Setups
  12. Result
  13. Learning with missing modalities
  14. Experimental Setup
  15. Evaluation Strategy
  16. ...and 33 more sections

Figures (3)

  • Figure 1: The framework of MLA: (a) Training stage, where we employ an alternating unimodal adaptation process, processing only one modality at each iteration. The shared head captures cross-modal information, and we also introduce gradient modification to prevent forgetting learned modality information from previous iterations; (b) Testing stage, where we introduce an uncertainty-based test-time multimodal fusion approach to combine multimodal information.
  • Figure 2: Visualization of test accuracy (%) on the KS dataset, varying with the ratio of missing modality in audio or video training data.
  • Figure 3: Visualizations of the modality gap distance on the Food101 dataset.