Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation

TL;DR

This work tackles synthetic multimodal data generation for radiology by adapting Composable Diffusion to chest X-ray imaging and report generation using MIMIC-CXR. They build a shared latent space via contrastive learning between frontal/lateral X-rays and radiology reports, train latent diffusion models for each modality, and enable cross-modal generation with additional cross-attention and alignment losses, formalized with $ \mathcal{L}_{A,B} $ and $ \mathcal{CL} = \mathcal{L}_{X,R} + \mathcal{L}_{R,X} $. Their results show improved FID and BLEU metrics, while generated images and reports achieve competitive or superior diagnostic fidelity on downstream tasks like disease classification (AUROC) and report-driven F1 via CheXpert labeling. The work demonstrates the practical potential of domain-specific synthetic multimodal data for privacy-preserving medical research and clinical support, highlighting the benefits of combining latent diffusion with contrastive cross-modal alignment.

Abstract

Generative models have revolutionized Artificial Intelligence (AI), particularly in multimodal applications. However, adapting these models to the medical domain poses unique challenges due to the complexity of medical data and the stringent need for clinical accuracy. In this work, we introduce a framework specifically designed for multimodal medical data generation. By enabling the generation of multi-view chest X-rays and their associated clinical report, it bridges the gap between general-purpose vision-language models and the specialized requirements of healthcare. Leveraging the MIMIC-CXR dataset, the proposed framework shows superior performance in generating high-fidelity images and semantically coherent reports. Our quantitative evaluation reveals significant results in terms of FID and BLEU scores, showcasing the quality of the generated data. Notably, our framework achieves comparable or even superior performance compared to real data on downstream disease classification tasks, underlining its potential as a tool for medical research and diagnostics. This study highlights the importance of domain-specific adaptations in enhancing the relevance and utility of generative models for clinical applications, paving the way for future advancements in synthetic multimodal medical data generation.

Figures (3)

• Figure 1: A sample of a frontal X-ray, a lateral X-ray and the corresponding radiology report.
• Figure 2: Framework of CoDi$_{\text{XR}}$: a) Shared Latent Space construction: Input modalities are processed by modality-specific encoders to extract feature representations, which are aligned in a shared latent space. b) Single modality generation: An LDM for each modality is trained to generate synthetic data from the latent representations extracted by the Prompt Encoders. c) Cross-modal alignment: it ensures consistency and alignment across jointly generated output modalities. Elements marked with a fire icon receive parameter updates during the specific training phase, while those marked with an ice icon remain frozen.
• Figure 3: Generation comparison between CoDi and $\text{CoDi}_{\mathit{{XR}}}$ using the same textual prompt, i.e., "X-ray presents no acute cardiopulmonary process".