MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

TL;DR

MultiBench introduces a large-scale, standardized benchmark for multimodal representation learning, spanning 15 datasets, 10 modalities, 20 tasks, and 6 research areas, paired with an end-to-end pipeline and a public leaderboard. It couples this benchmark with MultiZoo, a modular toolkit of 20 core multimodal methods covering data preprocessing, fusion paradigms, optimization objectives, and training procedures to enable reproducible, cross-domain evaluation. The study highlights that methodological transfer across domains can yield state-of-the-art results on several datasets while exposing critical tradeoffs between performance, training-time complexity, and robustness to imperfect data. Overall, MultiBench and MultiZoo provide a scalable, community-friendly platform to push toward generalizable, lightweight, and robust multimodal models across a diverse set of real-world modalities and tasks.

Abstract

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

Figures (26)

• Figure 1: MultiBench contains a diverse set of $15$ datasets spanning $10$ modalities and testing for more than $20$ prediction tasks across $6$ distinct research areas, thereby enabling standardized, reliable, and reproducible large-scale benchmarking of multimodal models. To reflect real-world requirements, MultiBench is designed to holistically evaluate (1) performance across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities.
• Figure 2: MultiZoo provides a standardized implementation of a suite of multimodal methods in a modular fashion to enable accessibility for new researchers, compositionality of approaches, and reproducibility of results.
• Figure 3: Tradeoff between performance and complexity. Size of circles shows variance in performance across (a) all datasets and (b) datasets on which we tested $>6$ approaches. We plot a dotted blue line of best quadratic fit to show the Pareto frontier. These strong tradeoffs should encourage future work in lightweight multimodal models that generalize across datasets, as well as in adapting several possibly well-performing methods (such as MFAS or MulT) to new datasets and domains.
• Figure 4: Tradeoff between performance and robustness. Size of circles shows variance in robustness across datasets. We show the line of best linear fit in dotted blue. While better performing methods show better relative robustness (a), some suffer in effective robustness since performance drops off faster (b). Few models currently achieve both relative and effective robustness, which suggests directions for future research.
• Figure 5: MultiBench provides a standardized machine learning pipeline across data processing, data loading, multimodal models, evaluation metrics, and a public leaderboard to encourage future research in multimodal representation learning. MultiBench aims to present a milestone in unifying disjoint efforts in multimodal machine learning research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility.