Balanced Multimodal Learning via Mutual Information
Rongrong Xie, Guido Sanguinetti
TL;DR
This work tackles modality imbalance in multimodal learning, focusing on biomedical multi-omics data. It introduces a unified framework that combines a revised Graph Convolutional Network (r-GCN) with edges derived from fused cross-modal similarities via similarity-network fusion (SNF), cross-modal knowledge distillation from strong to weak modalities, and a multitask-like optimization that dynamically balances gradient contributions using macro-$F_1$ and mutual information signals. Mutual information guides when knowledge transfer is plausible and informs fusion emphasis, while the multitask-like loss reweights unimodal and multimodal tasks to suppress dominant modalities and elevate weaker, informative channels. Applied to BRCA classification with CNV, mRNA, and RPPA data, the method yields macro-$F_1$ improvements over concatenation and unimodal baselines and remains robust under low-information channels and small sample regimes, suggesting a practical recipe for learning under scarcity and heterogeneity.
Abstract
Multimodal learning has increasingly become a focal point in research, primarily due to its ability to integrate complementary information from diverse modalities. Nevertheless, modality imbalance, stemming from factors such as insufficient data acquisition and disparities in data quality, has often been inadequately addressed. This issue is particularly prominent in biological data analysis, where datasets are frequently limited, costly to acquire, and inherently heterogeneous in quality. Conventional multimodal methodologies typically fall short in concurrently harnessing intermodal synergies and effectively resolving modality conflicts. In this study, we propose a novel unified framework explicitly designed to address modality imbalance by utilizing mutual information to quantify interactions between modalities. Our approach adopts a balanced multimodal learning strategy comprising two key stages: cross-modal knowledge distillation (KD) and a multitask-like training paradigm. During the cross-modal KD pretraining phase, stronger modalities are leveraged to enhance the predictive capabilities of weaker modalities. Subsequently, our primary training phase employs a multitask-like learning mechanism, dynamically calibrating gradient contributions based on modality-specific performance metrics and intermodal mutual information. This approach effectively alleviates modality imbalance, thereby significantly improving overall multimodal model performance.