Improving Multimodal Learning with Multi-Loss Gradient Modulation

TL;DR

This work improves upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence.

Abstract

Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, enhancing robustness, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue the vast majority of previous works suggest to assess the unimodal contributions and dynamically adjust the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We achieve superior results across three audio-video datasets: on CREMA-D, models with ResNet backbone encoders surpass the previous best by 1.9% to 12.4%, and Conformer backbone models deliver improvements ranging from 2.8% to 14.1% across different fusion methods. On AVE, improvements range from 2.7% to 7.7%, while on UCF101, gains reach up to 6.1%.

Figures (6)

• Figure 1: Categorization of state-of-the-art balancing methodology: (a) Gradient Balancing methods use estimates of unimodal performance to calculate coefficients ($k_a$ and $k_v$) and employ these to balance the multimodal loss. (b) Multi-Task methods incorporate unimodal classifiers into the model, each noted as CLS Head, to better estimate unimodal performance. The coefficients ($k_a$ and $k_v$) derived from comparing unimodal performance are used exclusively to balance the unimodal losses. (c) The proposed Multi-Loss Balanced method combines both strategies by incorporating unimodal classifiers for accurate unimodal performance estimation and balancing both multimodal and unimodal losses.
• Figure 2: Comparing Balancing Coefficients ($k_i$, Eq. \ref{['eq:ki']}) and Performance Ratios ($r_i$, Eq. \ref{['eq:ri']}) across different $\alpha$ and $\beta$ settings.
• Figure 3: Accuracy of models that differentiate by the backbone encoders (colors), the fusion strategies (Late with a linear and Mid with a MLP classifier), and the balancing techniques (x-axis). Across all datasets, results demonstrate that employing unimodal losses within the Multi-Loss framework and balancing them consistently yields the best performance.
• Figure 4: Confusion Matrices between the unimodal models and the multimodal ones for both backbone encoders, (a) Conformer and (b) ResNet, trained under different balancing methods. Each column of the confusion matrix represents the cases where both unimodal predictions are incorrect, where only one is correct, and where both are correct. MLB consistently balances and improves performance across all categories.
• Figure 5: ECE Comparison on Conformer CREMA-D.