Fusion or Confusion? Multimodal Complexity Is Not All You Need

TL;DR

This work addresses whether architectural complexity drives multimodal learning gains. It reimplements 19 contemporary methods across nine real-world datasets with up to 23 modalities under a standardized, rigorously tuned protocol, introducing SimBaMM, a simple late-fusion Transformer baseline. The study finds that complex architectures rarely outperform SimBaMM and often do not beat well-tuned unimodal baselines, highlighting the role of methodological rigor and strong unimodal encoders. It provides a reliability checklist and a case study illustrating evaluation pitfalls, advocating a shift from architectural novelty to robust, generalizable evaluation in multimodal research.

Abstract

Deep learning architectures for multimodal learning have increased in complexity, driven by the assumption that multimodal-specific methods improve performance. We challenge this assumption through a large-scale empirical study reimplementing 19 high-impact methods under standardized conditions, evaluating them across nine diverse datasets with up to 23 modalities, and testing their generalizability to new tasks beyond their original scope, including settings with missing modalities. We propose a Simple Baseline for Multimodal Learning (SimBaMM), a straightforward late-fusion Transformer architecture, and demonstrate that under standardized experimental conditions with rigorous hyperparameter tuning of all methods, more complex architectures do not reliably outperform SimBaMM. Statistical analysis indicates that more complex methods perform comparably to SimBaMM and frequently do not reliably outperform well-tuned unimodal baselines, especially in the small-data regime considered in many original studies. To support our findings, we include a case study of a recent multimodal learning method highlighting the methodological shortcomings in the literature. In addition, we provide a pragmatic reliability checklist to promote comparable, robust, and trustworthy future evaluations. In summary, we argue for a shift in focus: away from the pursuit of architectural novelty and toward methodological rigor.

Figures (5)

• Figure 1: Taxonomy of multimodal learning methods exemplified by two modalities. A late-fusion baseline (simbamm) processes a multimodal dataset $D$ with up to $M$ modalities through encoders $E_m$, combines them via fusion, and produces predictions through a classification head. Categorized into five groups, we evaluate 19.0 sota methods across nine datasets based on their architectural innovations on top of simbamm: Missing Data handling for incomplete modalities, Encoder-level modifications, Fusion mechanisms for combining representations, Head-based cross-modal architectures, and Gradient-based optimization techniques. Our large-scale empirical study reveals that none of these complex extensions reliably outperforms simbamm, suggesting that reported performance gains are often attributable to experimental confounders rather than methodological novelty.
• Figure 2: Missing-modality analysis on MIMIC Symile: methods perform similarly across missing rates, with overlapping sd.
• Figure 3: Efficiency analysis for training time (left), FLOPS (middle), and parameters (right) on the MIMIC Symile dataset. Regardless of the metric, simbamm and simbamm$^{\text{CLS}}$ perform consistently while others, e.g., Coupled Mamba and IMDer are metric-dependent.
• Figure 4: Case study on the CREMA-D dataset. Under a corrected data protocol with subject-independent splits and consistent hyperparameter tuning, the relative performance of methods changes substantially.
• Figure 5: Visual representation of the efficiency of the methods w.r.t. the number of parameters, training time, and flops.