I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts

TL;DR

I2MoE addresses the challenge of multimodal fusion by explicitly modeling heterogeneous inter-modal interactions within an end-to-end mixture-of-experts framework. It deploys four interaction experts (two uniques, one synergy, one redundancy) plus a reweighting module, trained with a dual objective that includes a weak supervision interaction loss inspired by Partial Information Decomposition, and extends naturally to more modalities. The approach yields superior task performance across five real-world multimodal datasets and provides both local (sample-level) and global (dataset-level) interpretability of interaction contributions. Its backbone-agnostic design allows easy integration with existing fusion methods, and code availability enables practical adoption and further exploration of modality interactions in multimodal learning.

Abstract

Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.

Figures (8)

• Figure 1: An illustrative example of modality interaction. The poster and plot are taken from the IMDB dataset.
• Figure 2: Comparison between vanilla modality fusion and I$^2$MoE in the case of movie genre classification with two input modalities. Left: Existing modality fusion approaches typically use the same parameters to model all types of interactions between the two modalities. Right: In contrast, we design a mixture-of-experts framework that employs four different interaction experts and a re-weighting model to explicitly capture heterogeneous interactions between the two input modalities.
• Figure 3: Qualitative example of local interpretation on the IMDB dataset provided by I$^2$MoE-MulT. Ground truth labels are Comedy, Adventure, Fantasy, Family, and Animation. (a) Logits output by different interaction experts. (b) Weighting assigned by the reweighting model. (c) Contribution of each interaction expert to the final weighted logit. (d) Raw image and language modalities used for prediction.
• Figure 4: Visualization of interaction weight distributions across all test samples for five datasets. Black bars indicate the median, mean, and extreme values.
• Figure 5: Comparison between the task performance of I$^2$MoE-MulT (red horizontal line) and each individual interaction expert across different datasets.