Cross Modality Medical Image Synthesis for Improving Liver Segmentation

TL;DR

This paper tackles data scarcity in medical image CAD by generating cross-modality MRI data from unpaired CT images using EssNet, an end-to-end network that combines CycleGAN-based translation with a segmentation-guided branch to overcome alignment issues. The synthetic MRI images, when combined with real MRI, improve liver segmentation performance of a U-Net, achieving an IoU gain of 1.17% and a Dice gain of 0.65%. Evaluations on CHAOS abdominal CT/MR data demonstrate the viability of cross-modality image synthesis for modest improvements in segmentation when labeled data are limited. Limitations include single-site data and evaluation restricted to U-Net, suggesting future work with broader datasets and advanced segmentation models.

Abstract

Deep learning-based computer-aided diagnosis (CAD) of medical images requires large datasets. However, the lack of large publicly available labeled datasets limits the development of deep learning-based CAD systems. Generative Adversarial Networks (GANs), in particular, CycleGAN, can be used to generate new cross-domain images without paired training data. However, most CycleGAN-based synthesis methods lack the potential to overcome alignment and asymmetry between the input and generated data. We propose a two-stage technique for the synthesis of abdominal MRI using cross-modality translation of abdominal CT. We show that the synthetic data can help improve the performance of the liver segmentation network. We increase the number of abdominal MRI images through cross-modality image transformation of unpaired CT images using a CycleGAN inspired deformation invariant network called EssNet. Subsequently, we combine the synthetic MRI images with the original MRI images and use them to improve the accuracy of the U-Net on a liver segmentation task. We train the U-Net on real MRI images and then on real and synthetic MRI images. Consequently, by comparing both scenarios, we achieve an improvement in the performance of U-Net. In summary, the improvement achieved in the Intersection over Union (IoU) is 1.17%. The results show potential to address the data scarcity challenge in medical imaging.

Figures (12)

• Figure 1: Our proposed two-stage (EssNet+U-Net) approach. Stage 1: The approach uses CycleGAN-based EssNet architecture to generate abdominal MRI. Stage 2: The generated MRI is combined with real MRI to improve the U-Net segmentation.
• Figure 2: Overall workflow of the EssNet architecture. Part 1 of the training module is the synthesis part, which is basically CycleGAN. $G_1$ and $G_2$ are the two generators, while $D_1$ and $D_2$ are the two discriminators. The segmentation part for end-to-end end training is in Part 2. In the testing module, real CT is fed to generate additional MRI images. (adapted from Huo et al. 23).
• Figure 3: The generator architecture includes an encoder block, a 9-block ResNet, and a decoder block.
• Figure 4: Architecture of a patch-based discriminator. The discriminator has five convolution blocks.
• Figure 5: Samples of abdominal CT images. These are the original CT images from the abdominal CT dataset and show different organs of the abdomen.