Vision-Language Synthetic Data Enhances Echocardiography Downstream Tasks

TL;DR

This work tackles the scarcity of annotated echocardiography data by adopting diffusion-based synthesis guided by vision–language signals. It introduces a latent-space diffusion framework with three conditioning modes—unconditional, text-guided, and text+segmentation-guided—to produce realistic echo images that preserve anatomical structures. The approach leverages CLIP encodings and ControlNet to incorporate semantic maps, yielding superior downstream segmentation and classification performance on CAMUS, and demonstrates clear advantages in realism and convergence. By releasing checkpoints, prompts, and a synthetic dataset, the paper offers a practical path to scalable, context-rich echo data generation with potential to accelerate clinical DL applications.

Abstract

High-quality, large-scale data is essential for robust deep learning models in medical applications, particularly ultrasound image analysis. Diffusion models facilitate high-fidelity medical image generation, reducing the costs associated with acquiring and annotating new images. This paper utilizes recent vision-language models to produce diverse and realistic synthetic echocardiography image data, preserving key features of the original images guided by textual and semantic label maps. Specifically, we investigate three potential avenues: unconditional generation, generation guided by text, and a hybrid approach incorporating both textual and semantic supervision. We show that the rich contextual information present in the synthesized data potentially enhances the accuracy and interpretability of downstream tasks, such as echocardiography segmentation and classification with improved metrics and faster convergence. Our implementation with checkpoints, prompts, and the created synthetic dataset will be publicly available at \href{https://github.com/Pooria90/DiffEcho}{GitHub}.

Figures (4)

• Figure 1: Overview of our proposed model. For both text and text+segmentation models, we use the CLIP text encoder radford2021learning while for the text+segmentation setting, the image encoder is a trainable copy for the denoising UNet.
• Figure 2: Three generation results from our text+segmentation model and SOTA SDM stojanovski2023echo. From the left in each row, the first and second columns indicate a sample output of the networks using the semantic map on the third column. The fourth column contains the ground truth. Green, red, and yellow boxes show the visual improvements achieved in the right chambers, mitral valve, and tricuspid valve, respectively.
• Figure 3: This figure showcases a selection of synthetic images generated by our models, illustrating various characteristics and outcomes. The unconditional model produced images with generally low brightness, particularly in the 2CH views, and some instances of anatomical mirroring can be observed in the 4CH-ES images (top row). The text-conditioned models, aligning with the previously reported FID scores in the paper, indicate poor performance in 2CH-ES results. However, they successfully depicted both the open and closed states of the mitral valve in the third and fourth columns of the middle row. Additionally, the text+segmentation model was distinguished by its generation of images with notably higher contrast, demonstrating the capabilities of our approaches in producing diverse, high-fidelity images.
• Figure 4: Illustration of some selected failure cases of our Real+100% segmentation model, highlighting specific challenges encountered during validation. In the top row, we observe a rare scenario from our validation set characterized by a small area of interest, where the model incorrectly identifies the entire surface of the Left Ventricle (LV) and Left Atrium (LA) as the LV endocardium in the predicted map. The second row illustrates a case of label confusion, where the LA label erroneously merges with the LV endocardium, leading to inaccurate segmentation. Finally, the third row shows a misguidance example, where a black circled area at the bottom of the LA has misled the model, resulting in a deviation from the correct LA label prediction. Upon examination of these phenomena, we concluded that the regions demonstrating failures are infrequently represented in the training set, which hinders the model's ability to properly interpret text or segmentation guidance. It is noteworthy to mention that our segmentation network employs a simple, lightweight UNet architecture as our main goal was just to demonstrate the potential of the synthesized data in enhancing the performance of downstream tasks. These instances underscore the complexity of accurately modeling cardiac structures and the potential for improvement in our segmentation approach.