Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Multimodal Paradox: How Added and Missing Modalities Shape Bias and Performance in Multimodal AI

Kishore Sampath, Pratheesh, Ayaazuddin Mohammad, Resmi Ramachandranpillai

TL;DR

This work examines how modality availability affects performance and fairness in multimodal AI for healthcare. It introduces a unified representation framework to fuse disparate modalities and evaluates both training-time addition and inference-time masking across two PhysioNet-derived datasets. Key findings show that adding modalities generally improves accuracy, but fairness outcomes are dataset-dependent, while missing modalities at deployment consistently reduce both performance and fairness. The study highlights the importance of fairness-aware design and robustness considerations for real-world multimodal systems.

Abstract

Multimodal learning, which integrates diverse data sources such as images, text, and structured data, has proven superior to unimodal counterparts in high-stakes decision-making. However, while performance gains remain the gold standard for evaluating multimodal systems, concerns around bias and robustness are frequently overlooked. In this context, this paper explores two key research questions (RQs): (i) RQ1 examines whether adding a modality con-sistently enhances performance and investigates its role in shaping fairness measures, assessing whether it mitigates or amplifies bias in multimodal models; (ii) RQ2 investigates the impact of missing modalities at inference time, analyzing how multimodal models generalize in terms of both performance and fairness. Our analysis reveals that incorporating new modalities during training consistently enhances the performance of multimodal models, while fairness trends exhibit variability across different evaluation measures and datasets. Additionally, the absence of modalities at inference degrades performance and fairness, raising concerns about its robustness in real-world deployment. We conduct extensive experiments using multimodal healthcare datasets containing images, time series, and structured information to validate our findings.

The Multimodal Paradox: How Added and Missing Modalities Shape Bias and Performance in Multimodal AI

TL;DR

This work examines how modality availability affects performance and fairness in multimodal AI for healthcare. It introduces a unified representation framework to fuse disparate modalities and evaluates both training-time addition and inference-time masking across two PhysioNet-derived datasets. Key findings show that adding modalities generally improves accuracy, but fairness outcomes are dataset-dependent, while missing modalities at deployment consistently reduce both performance and fairness. The study highlights the importance of fairness-aware design and robustness considerations for real-world multimodal systems.

Abstract

Multimodal learning, which integrates diverse data sources such as images, text, and structured data, has proven superior to unimodal counterparts in high-stakes decision-making. However, while performance gains remain the gold standard for evaluating multimodal systems, concerns around bias and robustness are frequently overlooked. In this context, this paper explores two key research questions (RQs): (i) RQ1 examines whether adding a modality con-sistently enhances performance and investigates its role in shaping fairness measures, assessing whether it mitigates or amplifies bias in multimodal models; (ii) RQ2 investigates the impact of missing modalities at inference time, analyzing how multimodal models generalize in terms of both performance and fairness. Our analysis reveals that incorporating new modalities during training consistently enhances the performance of multimodal models, while fairness trends exhibit variability across different evaluation measures and datasets. Additionally, the absence of modalities at inference degrades performance and fairness, raising concerns about its robustness in real-world deployment. We conduct extensive experiments using multimodal healthcare datasets containing images, time series, and structured information to validate our findings.
Paper Structure (11 sections, 3 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. RQ1: How does the compounding effect of introducing new modalities in training impact both the performance and bias of the model in downstream decision making?
  3. RQ2: How do missing modalities at inference impact performance and fairness of the model at deployment?
  4. Modality Integration and Prediction
  5. Experiments
  6. Settings
  7. Downstream tasks.
  8. Results
  9. RQ1
  10. RQ2
  11. Conclusion

Figures (3)

  • Figure 1: A multimodal framework for unified representation learning transforming heterogeneous clinical inputs (structured, time series and imaging data) into a common text-based feature space for downstream diagnostic classification tasks.
  • Figure 2: Impact of incrementally adding modalities during training on performance (left), DP (middle), and TPR (right). First row represents results for MIMIC-Eye and second row for MIMIC-IV-Ext-MDS-ED. X-axis represents Modalities sequentially added to the base ECG model and Y-axis represents metrics.
  • Figure 3: Model robustness to missing modalities during inference: performance (left), DP (middle), and TPR (right). First row represents results for MIMIC-Eye and second row for MIMIC-IV-Ext-MDS-ED. X-axis represents percentage of missing modalities and Y-axis represents metrics.