Using Synthetic Images to Augment Small Medical Image Datasets
Minh H. Vu, Lorenzo Tronchin, Tufve Nyholm, Tommy Löfstedt
TL;DR
This work tackles the problem of small medical image datasets by proposing a conditional StyleGAN2 framework to synthesize high-resolution, multi-modal medical images paired with segmentation masks. Three GAN variants are introduced: a baseline mask generator, a mask-only generator, and a conditional generator that produces image–mask pairs, with a pipeline that trains a U-Net on real data augmented by synthetic samples. Across six datasets, including an in-house pelvis dataset, results show that synthetic augmentation rarely improves segmentation performance, with significant gains only in two datasets and often no benefit or even harm when many synthetic samples are added. The study highlights the need for higher-fidelity generation and better alignment between synthetic and real data distributions to realize the potential of GAN-based augmentation in medical image analysis.
Abstract
Recent years have witnessed a growing academic and industrial interest in deep learning (DL) for medical imaging. To perform well, DL models require very large labeled datasets. However, most medical imaging datasets are small, with a limited number of annotated samples. The reason they are small is usually because delineating medical images is time-consuming and demanding for oncologists. There are various techniques that can be used to augment a dataset, for example, to apply affine transformations or elastic transformations to available images, or to add synthetic images generated by a Generative Adversarial Network (GAN). In this work, we have developed a novel conditional variant of a current GAN method, the StyleGAN2, to generate multi-modal high-resolution medical images with the purpose to augment small medical imaging datasets with these synthetic images. We use the synthetic and real images from six datasets to train models for the downstream task of semantic segmentation. The quality of the generated medical images and the effect of this augmentation on the segmentation performance were evaluated afterward. Finally, the results indicate that the downstream segmentation models did not benefit from the generated images. Further work and analyses are required to establish how this augmentation affects the segmentation performance.