Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory-Efficient 3D High-Resolution Medical Image Synthesis Using CRF-Guided GANs

Mahshid Shiri, Alessandro Bruno, Daniele Loiacono

TL;DR

This work tackles the memory bottleneck of 3D high-resolution medical image synthesis by introducing CRF-GAN, a memory-efficient architecture that enforces cross-patch consistency through a Conditional Random Field (CRF) applied to a mid-level image embedding. The generator is split into a two-step process (embedding generation and patch rendering from a subset), and a half-encoder bridges real and synthetic embeddings to train the CRF without extra networks. The CRF-GAN achieves better fidelity (lower FID and MMD) than the state-of-the-art HA-GAN at $256^3$, while also reducing model size, memory usage, and training time, and it benefits downstream augmentation tasks by improving recall and F1 in a nodule-detection setting. This approach offers a scalable path to high-quality 3D medical image synthesis with practical implications for data augmentation and clinical realism across CT and MRI domains.

Abstract

Generative Adversarial Networks (GANs) have many potential medical imaging applications. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models cannot scale to high-resolution or are susceptible to patchy artifacts. In this work, we propose an end-to-end novel GAN architecture that uses Conditional Random field (CRF) to model dependencies so that it can generate consistent 3D medical Images without exploiting memory. To achieve this purpose, the generator is divided into two parts during training, the first part produces an intermediate representation and CRF is applied to this intermediate representation to capture correlations. The second part of the generator produces a random sub-volume of image using a subset of the intermediate representation. This structure has two advantages: first, the correlations are modeled by using the features that the generator is trying to optimize. Second, the generator can generate full high-resolution images during inference. Experiments on Lung CTs and Brain MRIs show that our architecture outperforms state-of-the-art while it has lower memory usage and less complexity.

Memory-Efficient 3D High-Resolution Medical Image Synthesis Using CRF-Guided GANs

TL;DR

This work tackles the memory bottleneck of 3D high-resolution medical image synthesis by introducing CRF-GAN, a memory-efficient architecture that enforces cross-patch consistency through a Conditional Random Field (CRF) applied to a mid-level image embedding. The generator is split into a two-step process (embedding generation and patch rendering from a subset), and a half-encoder bridges real and synthetic embeddings to train the CRF without extra networks. The CRF-GAN achieves better fidelity (lower FID and MMD) than the state-of-the-art HA-GAN at , while also reducing model size, memory usage, and training time, and it benefits downstream augmentation tasks by improving recall and F1 in a nodule-detection setting. This approach offers a scalable path to high-quality 3D medical image synthesis with practical implications for data augmentation and clinical realism across CT and MRI domains.

Abstract

Generative Adversarial Networks (GANs) have many potential medical imaging applications. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models cannot scale to high-resolution or are susceptible to patchy artifacts. In this work, we propose an end-to-end novel GAN architecture that uses Conditional Random field (CRF) to model dependencies so that it can generate consistent 3D medical Images without exploiting memory. To achieve this purpose, the generator is divided into two parts during training, the first part produces an intermediate representation and CRF is applied to this intermediate representation to capture correlations. The second part of the generator produces a random sub-volume of image using a subset of the intermediate representation. This structure has two advantages: first, the correlations are modeled by using the features that the generator is trying to optimize. Second, the generator can generate full high-resolution images during inference. Experiments on Lung CTs and Brain MRIs show that our architecture outperforms state-of-the-art while it has lower memory usage and less complexity.

Paper Structure

This paper contains 16 sections, 5 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Conditional Random Field(CRF)
  3. Method
  4. Idea Behind Proposed Architecture
  5. Components
  6. Two-Step GAN Structure:
  7. Half-Encoder Structure:
  8. Proposed Architecture (CRF-GAN)
  9. Experiments
  10. Performance Analysis
  11. FID and MMD Scores:
  12. Simplicity:
  13. Memory Efficiency:
  14. Training Speed:
  15. Augmentation Task:
  16. ...and 1 more sections

Figures (7)

  • Figure 1: CRF-GAN architecture. CRF is applied to the embedding of real images(output of $E$) and synthetic images(output of $G_1$)
  • Figure : CRF-GAN synthetic images at different iterations
  • Figure : CRF-GAN synthetic images at different iterations
  • Figure : CRF-GAN synthetic images at different iterations
  • Figure : HA-GAN synthetic images at different iterations
  • ...and 2 more figures