Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Bang Zheng, Kyle Lam, Lin Li, Jianing Qiu

Abstract

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE ($\it{e.g.}$, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

Abstract

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE (, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.
Paper Structure (66 sections, 3 equations, 16 figures, 14 tables)

This paper contains 66 sections, 3 equations, 16 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models.
  4. Multimodal Medical Benchmarks.
  5. Benchmark
  6. Data Collection and Preprocessing
  7. Data Collection.
  8. Data Preprocessing.
  9. Benchmark Design
  10. DDx generation.
  11. FDx selection.
  12. Metrics.
  13. Experiments
  14. Evaluation
  15. Main Results
  16. ...and 51 more sections

Figures (16)

  • Figure 1: (a) Clinicians curate a broad differential diagnosis (DDx) list before determining a final diagnosis (FDx) via evidence synthesis. (b) Models exhibit a substantial gap between DDx coverage rate and FDx accuracy, far exceeding that observed in human experts.
  • Figure 2: Example final diagnosis selection tasks in English (top) and Chinese (bottom). Colors mark different visual clinical evidence (CE) types referenced in the question, with corresponding expert-derived diagnostic findings; gray denotes textual CE. In our experiment, each CE type is input as either raw images or text findings, not both.
  • Figure 3: (a) Distribution of visual clinical evidence (CE) types. (b) Number of visual CE types per case.
  • Figure 4: Mean final diagnosis accuracy (%) for proprietary models, domain-specific models, and open-source general-purpose models across clinical specialties.
  • Figure 5: Layer-wise Relative Attention per Token (RAPT) for Lingshu on text (excluding question stem) versus image tokens, before and after the Random-Text intervention.
  • ...and 11 more figures