Multimodal Classification via Modal-Aware Interactive Enhancement

TL;DR

This work tackles modality imbalance in multimodal learning by integrating Sharpness Aware Minimization (SAM) into a two-stage interaction framework. During forward passes, SAM smooths the learning objective to encourage flatter minima; during backward passes, a gradient-modification mechanism uses the flat directions from well-trained modalities to influence others, mitigating modality forgetting. The approach, termed modal-aware interactive enhancement (MIE), combines a SAM-based objective with an SVD-driven gradient routing matrix to promote cross-modal synergy. Empirical results across five diverse datasets show consistent improvements over state-of-the-art baselines, including stronger robustness with pretrained backbones such as CLIP.

Abstract

Due to the notorious modality imbalance problem, multimodal learning (MML) leads to the phenomenon of optimization imbalance, thus struggling to achieve satisfactory performance. Recently, some representative methods have been proposed to boost the performance, mainly focusing on adaptive adjusting the optimization of each modality to rebalance the learning speed of dominant and non-dominant modalities. To better facilitate the interaction of model information in multimodal learning, in this paper, we propose a novel multimodal learning method, called modal-aware interactive enhancement (MIE). Specifically, we first utilize an optimization strategy based on sharpness aware minimization (SAM) to smooth the learning objective during the forward phase. Then, with the help of the geometry property of SAM, we propose a gradient modification strategy to impose the influence between different modalities during the backward phase. Therefore, we can improve the generalization ability and alleviate the modality forgetting phenomenon simultaneously for multimodal learning. Extensive experiments on widely used datasets demonstrate that our proposed method can outperform various state-of-the-art baselines to achieve the best performance.

Figures (6)

• Figure 1: Comparison of different paradigms. (a). Multimodal learning paradigm. (b). The proposed paradigm in this paper. Compared with traditional MML paradigm, we design a SAM-based optimization strategy by introducing a perturbation parameters. We further utilize a gradient modification strategy to enhance the interaction between different modalities.
• Figure 2: The architecture of our proposed MIE. Our method contains two key components, i.e., SAM-based optimization (shown in the upper left corner of the panel) and interactive gradient modification (shown in the right part of the panel).
• Figure 3: Interactive enhancement analysis (best view in color).
• Figure 4: Sensitivity to $\tau$ and $\rho$.
• Figure 5: Loss landscape visualization. The first/second rows show the loss landscape for video/audio modalities, respectively.