M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation

TL;DR

M4CXR presents a multi-modal LLM for chest X-ray interpretation that integrates multi-image inputs and multiple tasks (medical report generation, visual grounding, and visual question answering) via a visual instruction-following dataset. It employs a chain-of-thought prompting strategy to improve clinical accuracy in report generation and demonstrates that multi-image and prior-study inputs enhance performance across scenarios. The model achieves competitive or state-of-the-art results in medical report generation, grounding, and VQA while maintaining end-to-end multi-task capabilities, validated on diverse datasets. The work highlights practical radiology automation potential, along with limitations such as NLG quality gaps and hallucinations, pointing to future improvements through dataset tailoring and prompting refinements.

Abstract

The rapid evolution of artificial intelligence, especially in large language models (LLMs), has significantly impacted various domains, including healthcare. In chest X-ray (CXR) analysis, previous studies have employed LLMs, but with limitations: either underutilizing the multi-tasking capabilities of LLMs or lacking clinical accuracy. This paper presents M4CXR, a multi-modal LLM designed to enhance CXR interpretation. The model is trained on a visual instruction-following dataset that integrates various task-specific datasets in a conversational format. As a result, the model supports multiple tasks such as medical report generation (MRG), visual grounding, and visual question answering (VQA). M4CXR achieves state-of-the-art clinical accuracy in MRG by employing a chain-of-thought prompting strategy, in which it identifies findings in CXR images and subsequently generates corresponding reports. The model is adaptable to various MRG scenarios depending on the available inputs, such as single-image, multi-image, and multi-study contexts. In addition to MRG, M4CXR performs visual grounding at a level comparable to specialized models and also demonstrates outstanding performance in VQA. Both quantitative and qualitative assessments reveal M4CXR's versatility in MRG, visual grounding, and VQA, while consistently maintaining clinical accuracy.

Figures (9)

• Figure 1: Overview of the multi-tasking capabilities of M4CXR. Facilitated by CoT prompting in MRG, M4CXR produces clinically accurate reports and adapts to various scenarios. Additionally, M4CXR can ground the locations described in the report or answer questions based on chest X-ray images.
• Figure 2: (a) The architecture of M4CXR. Utilizing the LLaVA framework, it allows visual tokens from each image to be inserted at designated positions among the text tokens. (b) Schema for constructing a CXR visual instruction-following dataset. Diverse tasks of three types are combined with appropriate sampling ratios.
• Figure 3: Example of multi-turn CoT prompting. M4CXR first identifies findings in the CXR image, then generates a report. Findings and corresponding sentences are color-matched for readability.
• Figure 4: Examples of M4CXR's performance in (a) visual grounding and (b) VQA. The images are selected from the test splits of MS-CXR and MIMIC-CXR, respectively.
• Figure 5: Examples of medical report generation across various scenarios. For the same study, the top left shows the result for single-image, the top right for multi-image, and the bottom for multi-study report generation. The results are selected from the MIMIC-CXR test set.