Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning

Alejandro Guerra-Manzanares, Farah E. Shamout

TL;DR

MILES addresses modality overfitting in multimodal learning by dynamically balancing learning across modalities using a modality-informed learning-rate scheduler. It defines epoch-wise conditional utilization rates and a delta imbalance measure to decide when and how to scale per-modality learning rates, adjusting only the dominant modality per epoch, and adds auxiliary supervision. Across four datasets and two fusion methods, MILES outperforms seven baselines in multimodal and unimodal metrics and yields stronger encoders for non-dominant modalities. The approach is simple, requires only a global learning rate and two hyperparameters, and demonstrates robust improvements and practical benefits for balanced multimodal learning with potential extensions to more modalities and missing data scenarios.

Abstract

The aim of multimodal neural networks is to combine diverse data sources, referred to as modalities, to achieve enhanced performance compared to relying on a single modality. However, training of multimodal networks is typically hindered by modality overfitting, where the network relies excessively on one of the available modalities. This often yields sub-optimal performance, hindering the potential of multimodal learning and resulting in marginal improvements relative to unimodal models. In this work, we present the Modality-Informed Learning ratE Scheduler (MILES) for training multimodal joint fusion models in a balanced manner. MILES leverages the differences in modality-wise conditional utilization rates during training to effectively balance multimodal learning. The learning rate is dynamically adjusted during training to balance the speed of learning from each modality by the multimodal model, aiming for enhanced performance in both multimodal and unimodal predictions. We extensively evaluate MILES on four multimodal joint fusion tasks and compare its performance to seven state-of-the-art baselines. Our results show that MILES outperforms all baselines across all tasks and fusion methods considered in our study, effectively balancing modality usage during training. This results in improved multimodal performance and stronger modality encoders, which can be leveraged when dealing with unimodal samples or absent modalities. Overall, our work highlights the impact of balancing multimodal learning on improving model performance.

MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning

TL;DR

MILES addresses modality overfitting in multimodal learning by dynamically balancing learning across modalities using a modality-informed learning-rate scheduler. It defines epoch-wise conditional utilization rates and a delta imbalance measure to decide when and how to scale per-modality learning rates, adjusting only the dominant modality per epoch, and adds auxiliary supervision. Across four datasets and two fusion methods, MILES outperforms seven baselines in multimodal and unimodal metrics and yields stronger encoders for non-dominant modalities. The approach is simple, requires only a global learning rate and two hyperparameters, and demonstrates robust improvements and practical benefits for balanced multimodal learning with potential extensions to more modalities and missing data scenarios.

Abstract

The aim of multimodal neural networks is to combine diverse data sources, referred to as modalities, to achieve enhanced performance compared to relying on a single modality. However, training of multimodal networks is typically hindered by modality overfitting, where the network relies excessively on one of the available modalities. This often yields sub-optimal performance, hindering the potential of multimodal learning and resulting in marginal improvements relative to unimodal models. In this work, we present the Modality-Informed Learning ratE Scheduler (MILES) for training multimodal joint fusion models in a balanced manner. MILES leverages the differences in modality-wise conditional utilization rates during training to effectively balance multimodal learning. The learning rate is dynamically adjusted during training to balance the speed of learning from each modality by the multimodal model, aiming for enhanced performance in both multimodal and unimodal predictions. We extensively evaluate MILES on four multimodal joint fusion tasks and compare its performance to seven state-of-the-art baselines. Our results show that MILES outperforms all baselines across all tasks and fusion methods considered in our study, effectively balancing modality usage during training. This results in improved multimodal performance and stronger modality encoders, which can be leveraged when dealing with unimodal samples or absent modalities. Overall, our work highlights the impact of balancing multimodal learning on improving model performance.
Paper Structure (21 sections, 5 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 21 sections, 5 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal joint fusion
  4. Modality competition
  5. Balanced multimodal learning
  6. Methodology
  7. Preliminaries
  8. Motivation
  9. Conditional utilization rate
  10. Modality-Informed Learning ratE Scheduler (MILES)
  11. Experiments
  12. Datasets
  13. Model training and evaluation
  14. Results
  15. Unimodal performance results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Validation accuracy of multimodal model and its unimodal encoders (left) and unimodal models trained independently (right) using the S-MNIST dataset.
  • Figure 2: Epoch-wise utilization metrics, learning rates, and validation accuracy for an example run on the CREMA-D dataset allow for the visualization of multimodal and unimodal learning dynamics using MILES.