Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering

TL;DR

This work introduces a novel Chest X-ray difference visual question answering task and the large-scale MIMIC-Diff-VQA dataset (164,324 image pairs and 700,703 QA pairs) to mimic radiologists’ practice of comparing current and past images. It proposes an anatomical-structure–aware feature extractor and an expert knowledge–aware multi-relational graph module, leveraging spatial, semantic, and implicit relations with a relation-aware graph attention network to produce accurate, interpretable answers for both disease-related and difference-focused questions. The approach outperforms baselines on non-difference VQA and demonstrates strong performance on difference questions, with ablations confirming the value of each graph type and evidence-fidelity analyses supporting interpretability. The work provides a foundation for clinically usable medical vision-language models and highlights opportunities for dataset expansion and improved feature backbones.

Abstract

To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Difference Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and, more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. We collect a new dataset, namely MIMIC-Diff-VQA, including 700,703 QA pairs from 164,324 pairs of main and reference images. Compared to existing medical VQA datasets, our questions are tailored to the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this task. The proposed baseline model leverages expert knowledge such as anatomical structure prior, semantic, and spatial knowledge to construct a multi-relationship graph, representing the image differences between two images for the image difference VQA task. The dataset and code can be found at https://github.com/Holipori/MIMIC-Diff-VQA. We believe this work would further push forward the medical vision language model.

Figures (7)

• Figure 1: (a) The ground truth report corresponding to the main(present) image. The red text represents labels incorrectly classified by either text mining or generated reports, while the red box marks the misclassified labels. The green box marks the correctly classified ones. The underlined text is correctly generated in the generated report. (b) The label "Pneumothorax" is incorrectly classified because there is NO evidence of pneumothorax from the chest X-ray. (c) "There is a new left apical pneumothorax" $\rightarrow$ This sentence is wrong because the evidence of pneumothorax was mostly improved after treatment. However, the vascular shadow in the left pulmonary apex is not very obvious, so it is understandable why it is misidentified as pneumothorax in the left pulmonary apex. "there is a small left pleural effusion" $\rightarrow$ It is hard for a doctor to tell if the left pleural effusion is present or not. (d) The ImageCLEF-VQA-Med questions are designed too simple. (e) The reference(past) image and clinical report. (f) Our medical difference VQA questions are designed to guide the model to focus on and localize important regions.
• Figure 2: Clinical motivation for Image difference VQA.
• Figure 3: Statistics by question types
• Figure 4: Structure of one study in the Key-Info dataset.
• Figure 5: Expert knowledge-aware image-difference graph for medical image difference visual question answering.