Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis

Mai A. Shaaban, Adnan Khan, Mohammad Yaqub

TL;DR

MedPromptX tackles the challenge of incomplete EHR data in chest X-ray diagnosis by fusing imaging and structured clinical information through a frozen Multimodal LLM, visual grounding, and dynamic few-shot prompting. The approach introduces DPS to refine few-shot candidates and VG to focus attention on relevant image regions, achieving state-of-the-art performance with an 11% improvement in F1-score over baselines. To support multimodal learning, the paper also presents MedPromptX-VQA, a dataset linking MIMIC-IV and MIMIC-CXR-JPG with 12 pathology labels across interleaved image-text records. The work demonstrates that grounding and adaptive prompt selection substantially enhance diagnostic accuracy while reducing the need for extensive fine-tuning, with code and data publicly available for replication and broader adoption in clinical settings.

Abstract

Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (EHR). This paper introduces MedPromptX, the first clinical decision support system that integrates multimodal large language models (MLLMs), few-shot prompting (FP) and visual grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis. A pre-trained MLLM is utilized to complement the missing EHR information, providing a comprehensive understanding of patients' medical history. Additionally, FP reduces the necessity for extensive training of MLLMs while effectively tackling the issue of hallucination. Nevertheless, the process of determining the optimal number of few-shot examples and selecting high-quality candidates can be burdensome, yet it profoundly influences model performance. Hence, we propose a new technique that dynamically refines few-shot data for real-time adjustment to new patient scenarios. Moreover, VG narrows the search area in X-ray images, thereby enhancing the identification of abnormalities. We also release MedPromptX-VQA, a new in-context visual question answering dataset encompassing interleaved images and EHR data derived from MIMIC-IV and MIMIC-CXR-JPG databases. Results demonstrate the SOTA performance of MedPromptX, achieving an 11% improvement in F1-score compared to the baselines. Code and data are publicly available on https://github.com/BioMedIA-MBZUAI/MedPromptX.

MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis

TL;DR

MedPromptX tackles the challenge of incomplete EHR data in chest X-ray diagnosis by fusing imaging and structured clinical information through a frozen Multimodal LLM, visual grounding, and dynamic few-shot prompting. The approach introduces DPS to refine few-shot candidates and VG to focus attention on relevant image regions, achieving state-of-the-art performance with an 11% improvement in F1-score over baselines. To support multimodal learning, the paper also presents MedPromptX-VQA, a dataset linking MIMIC-IV and MIMIC-CXR-JPG with 12 pathology labels across interleaved image-text records. The work demonstrates that grounding and adaptive prompt selection substantially enhance diagnostic accuracy while reducing the need for extensive fine-tuning, with code and data publicly available for replication and broader adoption in clinical settings.

Abstract

Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (EHR). This paper introduces MedPromptX, the first clinical decision support system that integrates multimodal large language models (MLLMs), few-shot prompting (FP) and visual grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis. A pre-trained MLLM is utilized to complement the missing EHR information, providing a comprehensive understanding of patients' medical history. Additionally, FP reduces the necessity for extensive training of MLLMs while effectively tackling the issue of hallucination. Nevertheless, the process of determining the optimal number of few-shot examples and selecting high-quality candidates can be burdensome, yet it profoundly influences model performance. Hence, we propose a new technique that dynamically refines few-shot data for real-time adjustment to new patient scenarios. Moreover, VG narrows the search area in X-ray images, thereby enhancing the identification of abnormalities. We also release MedPromptX-VQA, a new in-context visual question answering dataset encompassing interleaved images and EHR data derived from MIMIC-IV and MIMIC-CXR-JPG databases. Results demonstrate the SOTA performance of MedPromptX, achieving an 11% improvement in F1-score compared to the baselines. Code and data are publicly available on https://github.com/BioMedIA-MBZUAI/MedPromptX.
Paper Structure (12 sections, 2 equations, 3 figures, 6 tables)

This paper contains 12 sections, 2 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. MedPromptX for Diagnosis
  4. Visual Grounding (VG)
  5. Dynamic Proximity Selection (DPS)
  6. Multimodal LLM (MLLM)
  7. MedPromptX-VQA Dataset
  8. Experimental Setup
  9. Results and Discussion
  10. Ablations
  11. Conclusion
  12. Appendix

Figures (3)

  • Figure 1: MedPromptX: each input sample consists of an image $I'$ and corresponding text $T'$ containing tabular features. (1) The VG model takes $I'$ and generates a grounded image $G'$ by prompting the desired output. (2) The grounded image embeddings $G$ and text embeddings $T$ of the candidates are processed by the DPS technique to calculate their relevancy scores to a query sample $q$. (3) MLLM ingests a few-shot prompt and predicts whether a patient is likely to have a targeted disease.
  • Figure 2: The "Positive" and "Negative" representations for 12 pathological conditions.
  • Figure A.1: Comparison of MedPromptX under different DPS thresholds.