Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study

TL;DR

A model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping and outperformed other one-to-one translation models in quantitative evaluations is proposed.

Abstract

Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis providing complementary information for identifying clinically important biomarkers. However, in clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. In this paper, we propose a model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping. The 3D LDM combined with conditioning using the target modality allows generating high-quality target modality in 3D overcoming the shortcoming of the missing out-of-slice information in 2D generation methods. The switchable block, noted as multiple switchable spatially adaptive normalization (MS-SPADE), dynamically transforms source latents to the desired style of the target latents to help with the diffusion process. The MS-SPADE block allows us to have one single model to tackle many translation tasks of one source modality to various targets removing the need for many translation models for different scenarios. Our model exhibited successful image synthesis across different source-target modality scenarios and surpassed other models in quantitative evaluations tested on multi-modal brain magnetic resonance imaging datasets of four different modalities and an independent IXI dataset. Our model demonstrated successful image synthesis across various modalities even allowing for one-to-many modality translations. Furthermore, it outperformed other one-to-one translation models in quantitative evaluations.

Figures (8)

• Figure 1: Overview of the proposed image-to-image translation process based on latent diffusion model. Our model utilizes the proposed MS-SPADE to transform the latent representation into the target latents and enables image synthesis in the desired target modality through conditioning.
• Figure 2: The source and target images are 3D volume images and our method is applied in a 3D manner. In (a), we depict the autoencoder to compute source latents ($z^{src}$ ) and the SPADE block to dynamically convert source latents to target-like latents ($z^{src} \rightarrow z^{tar}_{src}$). The switchable block includes normalization layers that are applied differently depending on the target modality allowing translation to target multiple modalities within a single model. The output of the SPADE block is noted as target-like to emphasize the subsequent role of LDM to fully predict the target latents. The block is stacked N times. (b) illustrates the training process of the LDM model to predict the target latents $z^{tar}$ from the target-like latents $z_{src}^{tar}$ obtained from the SPADE block.
• Figure 3: (a) illustrates the training process of the autoencoder during the image compression phase to compute source latents, which involves five different losses: reconstruction, quantization, adversarial objective, perceptual, and cycle consistency. (b) shows the training process of the diffusion model, where the autoencoder is frozen, and only the UNet of the diffusion model is trained. The forward diffusion process is fixed and the input to the UNet consists of the concatenated two latents.
• Figure 4: The figures illustrate the results of the proposed model and comparison models on the BraTS2021 dataset for the image-to-image translation tasks (top: T1 $\rightarrow$ T2, bottom: T2 $\rightarrow$ FLAIR). These two tasks are representative tasks out of 12 possible tasks. The results are depicted in both the axial (top) and sagittal (bottom) views of the 3D volume. The green bounding boxes highlight the important tumor regions in the synthesized images providing a detailed visual representation.
• Figure 5: The figures illustrate the results of the proposed model and comparison models on the IXI dataset.