Quantifying Multimodal Imbalance: A GMM-Guided Adaptive Loss for Audio-Visual Learning

TL;DR

The paper tackles Multimodal Imbalance in audio-visual learning by introducing a quantitative Modality Gap metric and a Gaussian Mixture Model (GMM) to split samples into balanced and imbalanced groups. A two-stage training scheme, consisting of Warm-up and Adaptive Loss phases, uses GMM-derived posterior weights to apply targeted losses that minimize the overall modality gap and steer imbalanced samples toward the balanced distribution, with an annealing schedule to shift focus over time. Empirically, the approach achieves state-of-the-art accuracy on CREMA-D ($80.65\%$), AVE ($70.40\%$), and KineticSound ($72.42\%$), with further gains from fine-tuning high-quality samples selected by the GMM. This framework offers a principled, per-sample intervention for multimodal fusion, demonstrating strong practical impact for robust cross-modal learning.

Abstract

The heterogeneity of multimodal data leads to inconsistencies and imbalance, allowing a dominant modality to steer gradient updates. Existing solutions mainly focus on optimization- or data-based strategies but rarely exploit the information inherent in multimodal imbalance or conduct its quantitative analysis. To address this gap, we propose a novel quantitative analysis framework for Multimodal Imbalance and design a sample-level adaptive loss function. We define the Modality Gap as the Softmax score difference between modalities for the correct class and model its distribution using a bimodal Gaussian Mixture Model(GMM), representing balanced and imbalanced samples. Using Bayes' theorem, we estimate each sample's posterior probability of belonging to these two groups. Based on this, our adaptive loss (1) minimizes the overall Modality Gap, (2) aligns imbalanced samples with balanced ones, and (3) adaptively penalizes each according to its imbalance degree. A two-stage training strategy-warm-up and adaptive phases,yields state-of-the-art performance on CREMA-D (80.65%), AVE (70.40%), and KineticSound (72.42%). Fine-tuning with high-quality samples identified by the GMM further improves results, highlighting their value for effective multimodal fusion.

Figures (4)

• Figure 1: Architecture-two stage training:warm up and adaptive loss training
• Figure 2: Visualization of the GMM fitting of $\mathbf{g}$ on the CREMA-D dataset
• Figure 3: Changes in the Gaussian Mixture (GM) distribution during adaptive training (CREMA-D dataset)
• Figure 4: Adaptive Training Improves Unimodal and Multimodal Accuracy on the CREMA-D Dataset