Deformation-aware GAN for Medical Image Synthesis with Substantially Misaligned Pairs

TL;DR

This work tackles the problem of cross-modality medical image synthesis when training pairs are substantially misaligned, such as lung MRI-CT with respiratory motion. It introduces DA-GAN, a deformation-aware GAN that integrates symmetric registration with a multi-objective inverse-consistency framework and deformation-aware discriminators to jointly correct misalignment and improve image fidelity. The key contributions are the multi-level inverse-consistency loss L_mic and the deformation-aware adversarial loss L_adv_da, which are validated on a simulated brain misalignment dataset and a real lung MRI-CT dataset, outperforming eight state-of-the-art methods. The results demonstrate DA-GAN’s potential to enable accurate synthetic imaging for radiotherapy planning and other clinically relevant tasks under challenging misalignment conditions.

Abstract

Medical image synthesis generates additional imaging modalities that are costly, invasive or harmful to acquire, which helps to facilitate the clinical workflow. When training pairs are substantially misaligned (e.g., lung MRI-CT pairs with respiratory motion), accurate image synthesis remains a critical challenge. Recent works explored the directional registration module to adjust misalignment in generative adversarial networks (GANs); however, substantial misalignment will lead to 1) suboptimal data mapping caused by correspondence ambiguity, and 2) degraded image fidelity caused by morphology influence on discriminators. To address the challenges, we propose a novel Deformation-aware GAN (DA-GAN) to dynamically correct the misalignment during the image synthesis based on multi-objective inverse consistency. Specifically, in the generative process, three levels of inverse consistency cohesively optimise symmetric registration and image generation for improved correspondence. In the adversarial process, to further improve image fidelity under misalignment, we design deformation-aware discriminators to disentangle the mismatched spatial morphology from the judgement of image fidelity. Experimental results show that DA-GAN achieved superior performance on a public dataset with simulated misalignments and a real-world lung MRI-CT dataset with respiratory motion misalignment. The results indicate the potential for a wide range of medical image synthesis tasks such as radiotherapy planning.

Figures (6)

• Figure 1: Substantial misalignment in lung MRI-CT pairs due to respiratory motion.
• Figure 2: (a) Network architecture of DA-GAN. (b) $L_{mic}$ loss dynamically enhances image correspondence from three objectives. (c) $L_{adv\_da}$ loss guides discriminators to learn deformation for improved image fidelity.
• Figure 3: Visualisation of prediction and error images in the simulation experiments (NA-3).
• Figure 4: Visualization of synthesised images and error maps on the lung dataset. The blue arrows indicate our DA-GAN achieved better visual results at spine and heart.
• Figure 5: Example images with different levels of non-affine misalignment.