Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A 3D generative model of pathological multi-modal MR images and segmentations

Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Mark S. Graham, Tom Vercauteren, M. Jorge Cardoso

TL;DR

The paper addresses the scarcity of labelled 3D brain MRI data and the challenge of sharing pathology-rich datasets. It introduces brainSPADE3D, a two-stage conditional generator combining a latent diffusion model-based label generator with a 3D-SPADE image generator to produce multi-modal MR images and semantic maps conditioned on pathological phenotypes. Experiments demonstrate high-fidelity synthetic images and labels, the ability to generate unseen phenotypes, and improved WMH segmentation when tumours are present, particularly at $2mm^3$ isotropic resolution and with a $1mm^3$ variant. Code availability enables practical adoption and further development for synthetic data augmentation in clinical segmentation.

Abstract

Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.

A 3D generative model of pathological multi-modal MR images and segmentations

TL;DR

The paper addresses the scarcity of labelled 3D brain MRI data and the challenge of sharing pathology-rich datasets. It introduces brainSPADE3D, a two-stage conditional generator combining a latent diffusion model-based label generator with a 3D-SPADE image generator to produce multi-modal MR images and semantic maps conditioned on pathological phenotypes. Experiments demonstrate high-fidelity synthetic images and labels, the ability to generate unseen phenotypes, and improved WMH segmentation when tumours are present, particularly at isotropic resolution and with a variant. Code availability enables practical adoption and further development for synthetic data augmentation in clinical segmentation.

Abstract

Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.
Paper Structure (9 sections, 1 equation, 3 figures, 4 tables)

This paper contains 9 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Data
  4. Algorithm
  5. Downstream segmentation task
  6. Experiments
  7. Quality of the generated images and labels pairs
  8. WMH segmentation in the presence of tumour lesions
  9. Discussion & Conclusion

Figures (3)

  • Figure 1: Architecture of our two-stage model: the left block corresponds to the label generator, and the right block to the image generator. Training and inference pathways are differentiated with black, red and dashed arrows.
  • Figure 2: Example synthetic $1mm^3$ and $2mm^{3}$ isotropic labels and images generated using tumour+WMH (left) and WMH (right) conditioning. The augmented frame in the top left images shows the small WMH lesions near the ventricles.
  • Figure 3: Sample WMH predictions on the BRATS dataset (top) and the SABRE test set (bottom) for all our models, in red. The leftmost column shows the tumour mask for the BRATS dataset (in blue) and the ground truth WMH for SABRE.