Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation

Yannian Gu, Zhongzhen Huang, Linjie Mu, Xizhuo Zhang, Shaoting Zhang, Xiaofan Zhang

Abstract

Multimodal large language models (MLLMs) demonstrate considerable potential in clinical diagnostics, a domain that inherently requires synthesizing complex visual and textual data alongside consulting authoritative medical literature. However, existing benchmarks primarily evaluate MLLMs in end-to-end answering scenarios. This limits the ability to disentangle a model's foundational multimodal reasoning from its proficiency in evidence retrieval and application. We introduce the Clinical Understanding and Retrieval Evaluation (CURE) benchmark. Comprising $500$ multimodal clinical cases mapped to physician-cited reference literature, CURE evaluates reasoning and retrieval under controlled evidence settings to disentangle their respective contributions. We evaluate state-of-the-art MLLMs across distinct evidence-gathering paradigms in both closed-ended and open-ended diagnosis tasks. Evaluations reveal a stark dichotomy: while advanced models demonstrate clinical reasoning proficiency when supplied with physician reference evidence (achieving up to $73.4\%$ accuracy on differential diagnosis), their performance substantially declines (as low as $25.4\%$) when reliant on independent retrieval mechanisms. This disparity highlights the dual challenges of effectively integrating multimodal clinical evidence and retrieving precise supporting literature. CURE is publicly available at https://github.com/yanniangu/CURE.

CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation

Abstract

Multimodal large language models (MLLMs) demonstrate considerable potential in clinical diagnostics, a domain that inherently requires synthesizing complex visual and textual data alongside consulting authoritative medical literature. However, existing benchmarks primarily evaluate MLLMs in end-to-end answering scenarios. This limits the ability to disentangle a model's foundational multimodal reasoning from its proficiency in evidence retrieval and application. We introduce the Clinical Understanding and Retrieval Evaluation (CURE) benchmark. Comprising multimodal clinical cases mapped to physician-cited reference literature, CURE evaluates reasoning and retrieval under controlled evidence settings to disentangle their respective contributions. We evaluate state-of-the-art MLLMs across distinct evidence-gathering paradigms in both closed-ended and open-ended diagnosis tasks. Evaluations reveal a stark dichotomy: while advanced models demonstrate clinical reasoning proficiency when supplied with physician reference evidence (achieving up to accuracy on differential diagnosis), their performance substantially declines (as low as ) when reliant on independent retrieval mechanisms. This disparity highlights the dual challenges of effectively integrating multimodal clinical evidence and retrieving precise supporting literature. CURE is publicly available at https://github.com/yanniangu/CURE.
Paper Structure (14 sections, 3 figures, 2 tables)

This paper contains 14 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The CURE Benchmark
  3. Data Collection and Benchmark Setup
  4. Dataset Statistics
  5. Task Formulation: End-to-End Diagnostic Reasoning
  6. 1. Differential Diagnosis Selection:
  7. 2. Open-ended Diagnosis Prediction:
  8. Data Availability and Reproducibility
  9. Experiments
  10. Experimental Setup
  11. The Impact of Evidence Retrieval Paradigms
  12. Analysis of Retrieved Evidence Quality
  13. Impact of Input Modalities on Diagnostic Performance
  14. Conclusion

Figures (3)

  • Figure 1: CURE benchmark pipeline. Cases are paired with PubMed reference evidence, and MLLMs are evaluated under different retrieval paradigms to disentangle multimodal understanding from evidence-grounded diagnosis performance.
  • Figure 2: Faithfulness analysis across evidence conditions. (a) Distribution of accuracy × groundedness quadrants averaged over 16 models. (b) Per-model groundedness rate under each evidence condition.
  • Figure 3: Heatmap of TF-IDF cosine similarities comparing retrieved evidence (RAG and AR), clinical history (Hx), and physician-annotated reference evidence (PR). Panels are stratified by baseline correctness.