Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models

TL;DR

The paper addresses data scarcity and privacy in medical imaging by systematically evaluating text-conditioned diffusion for synthetic medical images. It compares fine-tuning large latent diffusion models with training compact domain-specific models, and introduces MSDM, a medical diffusion model that combines a VAE, a clinical text encoder, and cross-attention. Evaluation on two clinical domains, MedVQA-GI for endoscopy and ROCOv2 for radiology, includes text-diversity paraphrasing and LoRA analysis, augmented by human expert judgments. Findings show large models achieve higher fidelity, but the compact MSDM attains comparable quality at lower computational cost, offering a practical path for education and privacy-preserving data sharing while highlighting the need for task-specific evaluation and clinician-in-the-loop refinement.

Abstract

The generation of realistic medical images from text descriptions has significant potential to address data scarcity challenges in healthcare AI while preserving patient privacy. This paper presents a comprehensive study of text-to-image synthesis in the medical domain, comparing two distinct approaches: (1) fine-tuning large pre-trained latent diffusion models and (2) training small, domain-specific models. We introduce a novel model named MSDM, an optimized architecture based on Stable Diffusion that integrates a clinical text encoder, variational autoencoder, and cross-attention mechanisms to better align medical text prompts with generated images. Our study compares two approaches: fine-tuning large pre-trained models (FLUX, Kandinsky) versus training compact domain-specific models (MSDM). Evaluation across colonoscopy (MedVQA-GI) and radiology (ROCOv2) datasets reveals that while large models achieve higher fidelity, our optimized MSDM delivers comparable quality with lower computational costs. Quantitative metrics and qualitative evaluations by medical experts reveal strengths and limitations of each approach.

Figures (6)

• Figure 1: A simplified overview of the text-to-image diffusion model pipeline, commonly used in most modern architectures.
• Figure 2: Overview of Kandinsky 2.2 architecture.
• Figure 3: Examples of images generated using FLUX, Kandinsky 2.2 and MSDM compared to the original images from the development dataset.
• Figure 4: Examples of images generated using FLUX, Kandinsky 2.2 and MSDM compared to the original images from the ROCOv2 dataset.
• Figure 5: Human expert and quantitive metrics evaluation scores for each model across 40 test prompts.