Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges

TL;DR

This systematic review synthesizes recent advances in generative modeling for synthetic medical data across imaging, text, time-series, and tabular modalities, focusing on GANs, VAEs, Diffusion Models, and LLMs from 2021 to 2023. It dissects synthesis applications, generation technologies, and evaluation methods, revealing a broad but uneven landscape: clinically valid syntheses abound across modalities, yet personalization, robust clinical validation, and standardized evaluation remain underdeveloped. Conditional generation and text-guided approaches show promise, but many studies underutilize prior clinical knowledge and patient context, limiting realism and utility. The review concludes with practical recommendations for tailoring generation to data/tasks, expanding beyond augmentation to validation/testing, improving benchmarking, and strengthening privacy assessments to accelerate clinical adoption of synthetic data.

Abstract

This paper presents a comprehensive systematic review of generative models (GANs, VAEs, DMs, and LLMs) used to synthesize various medical data types, including imaging (dermoscopic, mammographic, ultrasound, CT, MRI, and X-ray), text, time-series, and tabular data (EHR). Unlike previous narrowly focused reviews, our study encompasses a broad array of medical data modalities and explores various generative models. Our search strategy queries databases such as Scopus, PubMed, and ArXiv, focusing on recent works from January 2021 to November 2023, excluding reviews and perspectives. This period emphasizes recent advancements beyond GANs, which have been extensively covered previously. The survey reveals insights from three key aspects: (1) Synthesis applications and purpose of synthesis, (2) generation techniques, and (3) evaluation methods. It highlights clinically valid synthesis applications, demonstrating the potential of synthetic data to tackle diverse clinical requirements. While conditional models incorporating class labels, segmentation masks and image translations are prevalent, there is a gap in utilizing prior clinical knowledge and patient-specific context, suggesting a need for more personalized synthesis approaches and emphasizing the importance of tailoring generative approaches to the unique characteristics of medical data. Additionally, there is a significant gap in using synthetic data beyond augmentation, such as for validation and evaluation of downstream medical AI models. The survey uncovers that the lack of standardized evaluation methodologies tailored to medical images is a barrier to clinical application, underscoring the need for in-depth evaluation approaches, benchmarking, and comparative studies to promote openness and collaboration.

Figures (12)

• Figure 1: Specialization trends in related survey papers- The papers are either focused on a single data type, or combine a single data type with one specific technology, or are exclusively dedicated to data augmentation applications.
• Figure 2: PRISMA flowchart of the literature screening process
• Figure 3: An overview of the number of surveyed papers per modality
• Figure 4: (a) The different data types and modalities covered in the survey. The different EHR formats also represent the synthesis applications of EHR. (b) Overview of synthesis applications of the imaging and signals data types. The left side focuses on imaging data, while the right side focuses on signals. An example of inter-modal synthesis involves the transformation of a brain CT scan to an MRI in imaging or an ECG signal into a synthetic PPG signal in the signals data type, which helps in scenarios where certain imaging or signal modalities might be unavailable. Intra-modal synthesis involves the conversion of a brain MRI T2 sequence into a T1 sequence or transforming a single lead into a 10 lead synthetic ECG signal. Attribute-conditioned synthesis shows a patient-specific brain MRI or an ECG signal being generated that matches attributes like age, BMI, ethnicity, and gender. Class or semantic map-conditioned synthesis shows a synthetic brain MRI with tumor being generated using a binary mask of a brain tumor or a synthetic ECG signal being generated based on class labels, such as C1 and C2. This can be useful for generating labeled datasets for training. The figure also illustrates text-guided synthesis, such as generating a synthetic chest X-ray based on a textual description like "moderate bilateral pleural effusions" or a synthetic ECG signal based on textual descriptions like "Left Bundle Branch Block". (c) Synthesis applications of the text data type
• Figure 5: Unconditional (left) vs conditional generation (right)