VLM-KG: Multimodal Radiology Knowledge Graph Generation

TL;DR

The paper tackles the challenge of generating radiology knowledge graphs from multimodal data (radiology reports and radiographs) by introducing VLM-KG, a framework that combines instruction tuning of a large language model with visual instruction tuning via a MedCLIP-based projector. This approach enables long-context processing ($32K$ tokens) and aligns visual and textual information to produce accurate KG triplets. Evaluated on MIMIC-CXR and IU-Xray RadGraph-derived datasets, VLM-KG achieves state-of-the-art BLEU scores and competitive ROUGE-L, with larger gains when trained on the larger MIMIC data, demonstrating the value of multimodal signals in reducing hallucinations. The work also releases the generated radiology KG dataset, highlighting practical implications for downstream radiology tasks and knowledge extraction pipelines.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable success in natural language generation, excelling at instruction following and structured output generation. Knowledge graphs play a crucial role in radiology, serving as valuable sources of factual information and enhancing various downstream tasks. However, generating radiology-specific knowledge graphs presents significant challenges due to the specialized language of radiology reports and the limited availability of domain-specific data. Existing solutions are predominantly unimodal, meaning they generate knowledge graphs only from radiology reports while excluding radiographic images. Additionally, they struggle with long-form radiology data due to limited context length. To address these limitations, we propose a novel multimodal VLM-based framework for knowledge graph generation in radiology. Our approach outperforms previous methods and introduces the first multimodal solution for radiology knowledge graph generation.

Figures (2)

• Figure 1: Overview of visual instruction tuning based multimodal radiology knowledge graph generation framework. The instruction dataset format is shown(Top). The Projector and the LLM are trained on the visual input and the instruction data for multimodal knowledge graph generation.
• Figure 2: Comparision of our multimodal VLM-KG models with unimodal Dygiee++ model for the task of knowledge graph generation. In the case of our models, both the radiographic image and the report in instruction format as shown in Figure \ref{['fig1']} are passed for the knowledge graph triplets generation. In the case of Dygiee++, which is an unimodal BERT-based model, only the radiology report is passed. The black color highlights the correct triplets corresponding to the radiology report. The red-colored triplets are incorrect and hallucinated predictions.