Asymmetric Reinforcing against Multi-modal Representation Bias

TL;DR

The paper tackles multimodal representation bias caused by shifting modality dominance in real-world settings. It introduces ARM, an asymmetric reinforcement framework that uses mutual information (MI) and conditional MI (CMI) to quantify each modality’s marginal and joint contributions, and to dynamically reinforce weaker modalities while preserving dominant ones. ARM combines dynamic feature-level fusion, a balanced min-max loss, and dynamic sample-level re-sampling to adapt to changing modality importance, backed by theoretical guarantees (Theorem 1) and extensive experiments on KS, UCF-51, and Food-101 showing improved accuracy and reduced modality-forgetting. The work advances robust multimodal learning by explicitly balancing cross-modal contributions, enabling better exploitation of inter-modal information and stronger generalization in imbalanced settings.

Abstract

The strength of multimodal learning lies in its ability to integrate information from various sources, providing rich and comprehensive insights. However, in real-world scenarios, multi-modal systems often face the challenge of dynamic modality contributions, the dominance of different modalities may change with the environments, leading to suboptimal performance in multimodal learning. Current methods mainly enhance weak modalities to balance multimodal representation bias, which inevitably optimizes from a partialmodality perspective, easily leading to performance descending for dominant modalities. To address this problem, we propose an Asymmetric Reinforcing method against Multimodal representation bias (ARM). Our ARM dynamically reinforces the weak modalities while maintaining the ability to represent dominant modalities through conditional mutual information. Moreover, we provide an in-depth analysis that optimizing certain modalities could cause information loss and prevent leveraging the full advantages of multimodal data. By exploring the dominance and narrowing the contribution gaps between modalities, we have significantly improved the performance of multimodal learning, making notable progress in mitigating imbalanced multimodal learning.

Figures (9)

• Figure 1: Accuracy curve of dominant modality compared with joint training baseline of imbalanced multimodal learning methods on Kinetics Sounds dataset. Other methods: Greedy greedy, AGMagm, Sample-valuation sample.
• Figure 2: Left: The Lower Bound joint contribution (MIV-LB) of all modalities and the Asymmetric marginal contribution (MIV-Asym) of each modality are estimated by $\phi^{MI}$ and $\phi^{CMI}$, respectively, serving as the basis for asymmetric reinforcement. $f_\mathcal{Y}$ is feature-level fusion result, $p$ is the accurate production. Right: Representation of features in the latent space. We minimize the diversities in $\phi^{CMI}$ to balance multimodal learning while maximizing $\phi^{MI}$ to enhance multimodal performance.
• Figure 3: Comparison of the narrowing trend of uni-modality contribution gaps on the UCF-51 dataset.
• Figure 4: Average joint contribution of all modalities overall training samples during training for Greedy, MLA, Sample-valuation, and our ARM on the KS and UCF-51 datasets.
• Figure 5: Curve of Balanced Min-Max Loss: the values are obtained from 5 training processes with the same initiations.