MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression

TL;DR

MMXU introduces a novel multi-image MedVQA benchmark focused on region-level disease progression across two chest X-ray visits, derived from MIMIC-CXR. The study demonstrates a persistent gap between state-of-the-art LVLMs and human radiologists on progression questions and addresses this with MedRecord-Augmented Generation (MAG), which injects global and regional historical records into model prompts. MAG yields significant accuracy gains across model types, with regional records frequently providing the strongest improvements, and benefits further when used to fine-tune models on MMXU-dev, narrowing the gap to expert performance. The work highlights the critical role of historical context in medical image interpretation and provides a publicly released dataset (MMXU) and MAG framework to spur further research in clinically grounded, temporally aware LVLMs.

Abstract

Large vision-language models (LVLMs) have shown great promise in medical applications, particularly in visual question answering (MedVQA) and diagnosis from medical images. However, existing datasets and models often fail to consider critical aspects of medical diagnostics, such as the integration of historical records and the analysis of disease progression over time. In this paper, we introduce MMXU (Multimodal and MultiX-ray Understanding), a novel dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. Unlike previous datasets that primarily address single-image questions, MMXU enables multi-image questions, incorporating both current and historical patient data. We demonstrate the limitations of current LVLMs in identifying disease progression on MMXU-\textit{test}, even those that perform well on traditional benchmarks. To address this, we propose a MedRecord-Augmented Generation (MAG) approach, incorporating both global and regional historical records. Our experiments show that integrating historical records significantly enhances diagnostic accuracy by at least 20\%, bridging the gap between current LVLMs and human expert performance. Additionally, we fine-tune models with MAG on MMXU-\textit{dev}, which demonstrates notable improvements. We hope this work could illuminate the avenue of advancing the use of LVLMs in medical diagnostics by emphasizing the importance of historical context in interpreting medical images. Our dataset is released at github: https://github.com/linjiemu/MMXU.

Figures (16)

• Figure 1: The upper part of the figure presents an example from our constructed dataset, which includes two images and a question regarding their changes. The lower part illustrates the performance of human experts and several models on three types of disease progression questions, emphasizing a notable gap between the models and human experts.
• Figure 2: The pipeline of constructing MMXU. In the first stage, we extract all comparative sentences and their associated labels from the Chest ImaGenome dataset. In the second stage, we refine our selection by applying precise filtering rules and identifying regions of interest within these sentences. The third stage involves leveraging GPT-4o with meticulously crafted prompts to generate region-level question-answer pairs. In the final stage, we further refine and filter generated QA pairs, thereby constructing the MMXU-test and MMXU-dev.
• Figure 3: Text template during the data post-processing.
• Figure 4: Two Examples from MMXU answered by some LVLMs. The left image shows successful cases with medical records enhancing the answers. The right image shows failed cases where historical records mislead the answers.
• Figure 5: Figure (a) shows the distribution of the number of patients corresponding to different study counts. We can see that a large number of patients have only one study. Figure (b) shows the blue line representing the changes between two studies of the same patient in MIMIC-CXR as the study order increases. The red dashed lines indicate the differences across patients with a number of studies no less than specific values.